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ABSTRACT 

The effects of Computer-Assisted Teacher Training 
System (CATTS) feedback in a preservice special educaticr teacher 
training program are discussed. It is explained that a series of 
studies were conducted to test the efficacy of CillS feedtack in 
effecting teacher trainees' acquisition and perfcrmance cf specific 
teaching skills. Chapter 1 presents the goals and objectives of the 
proiectr an overview of the project (pilot procedures^ descrimination 
traininqr laboratory classroom design, trainee evaluatict), and 
describes both the organization of the CATTS project and an overview 
of CATTS itself (teaching station, observation coding station, 
analysis-eDCoding station). Chapter 2 deals with the two cbservaticn 
systems used for feedback of teacher and student behaviors to 
trainees, COG-STRAT (focused on teacher and student cognitive styles 
of interaction) and MAN-STRAT (focused cd student on- and off-task 
behavior and strategies of teacher maDage»eDt of student tehavior), 
and discusses such areas as observer traiDiDg and evaluaticD of 
observer competencies. Such aspects of prcject crganization and 
implementation as teacher education laloratcry classrccKS, 
implementation procedures (teaching atd ceding) , scheduling 
observation and teachers, and the effects cf feedback on trainee 
behaviors are studied in chapter 3. Chapter 4 suEmarizes the results 
of the study which revealed that all tiainees signif icartly increased 
their rate of criterion performance as a furctiCD of CATTS feedtack. 
The summary and conclusions of the prcject are presented in the final 
chapter. (BD) 
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I. GO;\LS A.^^D OBJECTIVES OF THE STUDY AM) A.N OVERVIEW OF TllE PROJECT 

The present series of studies were conducted test the efficacy of 
Computer- Assisted Teacher Training System (CATTS) feedback in effecting teacher 
trainees' acquisition and performance of specific teaching skills. The 
realization of the major goal of trainee skill developrcent through CATTS 
entailed the continuing development of coinputer hardware and software 
support systems for real-time processing and feedback of specific observation 
coding data. A corsputer-based system for the simultaneous collection, pro- 
cessing and feedback of coded observational data from several classrooms 
a - t onc e was developed. In addition, realization of the training objectives 
of CATTS required the development of an efficient, reliable observer training 
program. For this purpose, a consensus coder training package bearing the 
acronynni *'DITR.^3A" was designed and integrated into the total effort aimed at 
improving; teacher-trainee skill performance - 

The continuing development of CATTS as a valid performar and cost- 
effective program for training texhers of handicapped childre- rested on 
several major assumptions: 

1 . The use of observation systems for training both supervisory per- 
sonnel and teachers to discriminate relevant teacher-learner behaviors 
provides an obj ective , common set of operational definition^ of the objec- 
tives of a given tericher-training program . The CATTS softw/ar-e system has 
been develcrr.-d to accommodate any set of observational dat,a smd iir-, , in 
past 2nplL'C3.tions, 'rje-i^n programmed to process data obta:::ne'^. th-r?v,;^h the 
Flander's -ys'ter^ Ifceider, 1969; Schmitt, 1969; Weaver, 19^(^9)^ z \q Individual 
Cognitive Demr -V>'5r.^^ni, ICDS, (Lynch 6 Ames, 1971), and th- ^ ^vior 

Manageme t ^ l^^, (Fink 5 Semmel , 1971). The ap^ i CATTo to 

the iiuisi-Mon teaching sV.ills rests in all instances l trainers' 
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detailed specificatior* of the objectives of the training prc^rar. through the 
niedium of a reliable observ'ation syster.. 

The use of ^.bser-^ation system coding for teacher training has in recent 
years gained recognition as a useful n:ethod of joining training objectives 
to a reliable measure of the trainee* 5 attainnient of perfonnance objectives. 
Originally developed as a method for describing classroom interaction 
CAnderso^ , 192>:', Smith § Meux, 1962) and for collecting such data for re- 
search purposes, the -system also has great utility as a method for focusing* 
trainees' attention on the specific interactional and classroom climate 
variables that are central to the given training progran. In the review of 
research on observation system studies in teacher education which accompanies 
this report, Semmel (1975) held that, although rhe-«^e are numerous and diverse 
observation systems available for coding teaching-learn' ' \ vav ors , 
procedures used in applying these systems in training ^ '^•-m ire c --^cv 

11> similar. The standard paradigm requires an : r iin ' e classrc' ^ 

tiD record the ongoing interactions according to a pre.'. tbed ccd.. Ttue 
._:assroom interaction may also be recorded on audio video ^ and latezr 
•^inscribed and coded to conform to the system emplc--:^d.. 'Tine cx cd drita ii-^ 
m analyzed and reported, usually as a summary of in cr.itsgor or 

f: e:!uency of occurrence in a category of behavior, g is a matrix ref?f^ct-ng 
the 5um of double-entry dyads (e.g., the number of items of behavior ca^s^ory 
X that followed category y) . The data is then transmitted back to the 
trainee for comparison of his performance with the perforr-ance critericr 
set by the training program. 

A limitation inherent in the typical training paradigm described .^...-i^^e 
is the delay of feedback of information to the trainee due to the time 
lag between collection of data and feedback to trainee necessitated by tiie 
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analysis processes. The CATTS syste:::: has been developed to overcome this 
limitation of observation systems for teacher training through the use of 
a coding system which utilizes the flexibility and speed of the computer to 
store and analyze coded data in real-tine, so that information on classroom 
interaction may be fed back to the trainee while he/she is still engaged in 
the interactive teaching process (Seinmel , 19''5) . 

It may be argued that the inureascd jsc . lil^.eo tape feedback in 
teacher training (Allen & Ryan, I:-o9, Hcrg-, et . 19701 mitigates ^ meces- 
sity for coding trainee's behavior ' >wc /er in cri .:al -r-v- e-. of fe^ed- 
back in teacher-training program^r . " ('rJ''^} cjrr^'' ^^f/sjoc 

that feedback which lacks a s" ific focu^ : eei: faizmd t-o diisngTe 

•.rainee behavior. They stres::-^*!:! the import-TiTi cueing -inec-fic behaivr~n*5, 

particularly in video tape (m:. crctea.: /iing) .it.... ^zonfrontiitl-on trainir"^ - 

echniqiues . 

On the ouher hand. Peck and Tucker Cl^"l -^'^ ladec thr feiedback is 
instrumental in changing teacher behavior cm: wh z:^: amothicv -^e-^r^on (e.g., 
supervisor or peer) participates in th e feedt L.jk session. Bn\wever, such 
performance change, if devoid of objectiveliy ijeasured predetermined ^i_ ,eria 
(such as provided by an observation system) , may be instrumental in changing 
trainees' behavior as interpreted by the supervir-or (e.g., subjectively), 
but would not offer any assurance thnt performamce goal^ of the trainmij 
program are met. 

2 . The second major assumption of the ZATTS training program is that , 
i n order for trainees to generate the desired inrreractive teaching behavi or? 
in the classroom, information on status of Lhe p^rrformance must be fed back 
to the trainee while he/she is still teaching . 'Hhc importance o''" feedback or 
knowledge of results in learning is a venerable axiom in the field of human 
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learning. In ceacher-training prograns geared to observation 5yster;S and 
in jnicroteaching, the principle of feedback and knowledge of results is 
central. However, the task of applying feedback principles, so potent in 
no tor skilJ and verbal 1 framing in t±ie laboratory, to skill and attitude 
developnent •^^.k:;her or counselor t v?. ining , is f ZTnidiiL' le. Not the lea/it 
of these prctienb- ^> the definition of immediate" fee ibvack. The typical 
teacher-training .7,rJv>gT Tjn is based upojii teach, supervisor)' feedback or 
video play^back :and rrr^:iich model (AI .cti. 1969; Borg 197C1). If the 
supervisor emp" s ar. v:iiective asse "','t instrumerr iich as an observa- 
tional system, *. en the delay in feedibac^>: may often be^ ;veral days or 
more. The CAT" ^ vstm is designed o '-ercome this ir^jcir limitation of 
the teach-fen-Uu^ C'-re teach strategy ±n reacher-training programs . The 
feedback avr«iL.;oie though CATTS is retur ^.ed to che cla.5sroom in real-time 
and is alsc ^tc red fcr retrieval for ; : -teaching 3inal> : is and inteipreta- 
tion. 

3. The setting cf teaching per'forrn.ance goals can be individualized so 
that each trainee may specify his own performance goals , Individiial per- 
formance goals can be determined on a logical basis if the trainee has 
access to a baseline of his own behaviors on previous lessons. Those 
categories of behavior where the individual performance on s given dimen- 
sion (e.g. , a single behavioral category or constellation of behavioral 
categories) falls above or below a desirable criterion may be selected as 
the individual's performance goal. 

The facility of a training program to individualize performance objec- 
tives is a vital one, as the program must account for individual differences 
in the entry level of trainees. Also, there is little evidence that optimal 
teaching behaviors are unidirectional or linear (see Medley 5 Soar, 1975; 



Semei, Seme! 5 Morrissey, 1976}. Thus, a training paradigm which involves 
simple and continuous increases in certain desirable teaching behaviors (e.g., 
indirect teacher questions, positive reinforcement) riay lack validity. A 
performance objective, such as the increase in positive reinforcenent of pupil 
responses, should be conditioned upon knowledge of the trainee *s level (pre- 
training) of positive responses to pupils. In cases where the trainee's entry 
level is high, further increases nay decrease tlie effectiveness of such be- 
havior. Tnus, a traini.ng program must be able to adjust general performance 
goals to the individual differences amonj: trainees. 

To sucmiarize the major assumption underlying the application of CATTS 
to teacher training; the trainees acquisition of teaching skills is viewed 
as dependent upon several major conditions; first, the trainees development 
of accuracy in the discrimination of teacher-learning behaviors; second, 
availability of reliable feedback on teaching-learning behaviors to the 
trainee during the course of an ongoing lesson; and third, the availability 
of valid and reliable data on the trainee's own performance over rime, for 
post-teaching analysis and review. 

Overview of the Project 

Pilot procedures . Prior to the initiation of the training program 
described in this report, several pilot projects were carried out during the 
previous school year at the Teacher Education Laboratory of The Center for 
Innovation in Teaching the Handicapped (CITH) to test the operating efficiency 
and the internal integrity of CATTS, as well as to determine the effective- 
ness of the system in instrumental ly improving the teaching performance of 
special education teacher-trainees. In order to evaluate the CATTS config- 
uration in teacher training, two initial pilot studies were carried out during 
the previous school year. The first study was derigned to ascertain the effect 
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of two t>'pes of CATTS feedback on increasing the nunbcr of high-level questions 
asked by the trainee during a given lesson. 'The two types of CATTS feedback 
studied were instant video display in the classroon, compared with delayed 
con^juter printout feedback following a lesson. 

The second pilot study compared the effect of the sane feedback variables 
on trainee questioning skills in a sir.ulated classroon setting. 

The experience obtained by the CATTS developi^ent team in design and 
execution of the earlier pilot studies was incorporated into a full test of 
the effects of the two t^^es of CATT? feedback on trainee behavior. A descrip- 
tion of the CATTS pilot studies is presented in Appendix I of this report. 

The experience gained from these initial attempts at iiriplenienting a 
teacher-training program under the CATT(S) system was reflected in the design 
of the present CATTS study in which three major areas of development and 
research were focal: (1) a coder-training (discrimination-training) program 
for trainees, (2) establ ishnent of a laboratory classroom, (3) a formative 
evaluation system and assessment of the effects of two types of feedback on 
trainee generation of prescribed teaching behaviors. 

Discrimination training . Both the pilot test phase of the CATTS project 
and a continuing review of the literature strongly indicated that the ability 
of the trainee to define and discriminate the critical classrooi interactive 
behaviors was a necessary precondition for the trainee to subsequently generate 
these behaviors during teaching. The CATTS project emphasized development of 
discrimination skills through a series af training modules designed to rein- 
force the objectives of CATTS training at each stage of implementation. To 
this end, a coder-training program was also developed, aimed ut establishing 
a high degree of coding: reliability on the part of the participant trainees. 
Coding reliability was assumed to be prim a facie evidence that the trainee 
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could discriminate the releyant behaviors of the systez^. 

The laboratory classroop . Several tv^pes of trainee support systens 
were designed and established to assist the trainees in lesson planning and 
pupil evaluation. Procedures for" pupil ^.election, scheduling, and trainee 
consultation were also developed for the operation of the laboratory class- 
room. 

c 

Trainee evaluation. A systen of formative evaluation of individual 
training goals was designed to provide trainees with an opportunity to 
measure and record their okti teaching behaviors and to adjust their inter- 
active patterns and goals, based on data made available through CATT5 delayed 
(printout) feedback . 

In addition to the enpirical determination of the efficacy of the two 
types of CATTS feedback on the trainees' acquisition of specific teaching 
skills,- several sunniative evaluation n^easures were taken, including trainee 
attitudes toward the program and their knowledge of t.^aching behaviors. 

The main objectives of the study of the use of CATTS in special educa- 
tion preservice teacher training were as follows: 

1. To develop software systems for real-time feedback of per- 
formance data to trainees while they are teaching, i 

2. To compare the effectiveness of two modes of feedback on trainee 
performance . 

3. To evaluate the CATTS model of preservice teacher preparation. 

O rganization of the CATTS Projec t 

Since the present project combined aspects of both an experimental 
study and that of a practicum course, coordination of a n'jonber of distinct 
operations was required. There were five major operations that characterized 
the project: (1) hardware and software development, (2) training of observers 
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in the discrimination of teaching behaviors, (3) -organization of a labori- 
tory classroom, (4) genei'atioi) of specific teaching behaviors with and j 
without CATTS feedback, and (5) evaluation of trainee performance. 

A detailed report on the CATTS configuration may be found in the 
CATTS Technical Reports (Semmel 5 Olson, 1977), while a brief summary of 
the hardware and computer software systems underlying the CATTS program 
is presented in the next section of this report. 

Chapter II presents a detailed description of the observation systems 
from which trainee performance objectives were derived. It also describes 
the development of materials and procedures for training teachers to become 
skilled, reliable coders of the observation system categories. 

The various phases in the development and implementation of the CA'iTS 
training program and research study are described in Chapter III. 

O verview of CATTS 

CATTS configuration . The system configuration of CATTS consists of 
three interdependent stations: Teaching Station , Observation-Coding 
Station, and Analysis-Encoding Station. Figures 1 and 2 illustrate .this 
configuration with schematic diagrams of the Present CATTS installation at 
the Teacher Education Laboratory- (TEL) , at CITH. 

Teaching Station . The CATTS Teaching :iLation consists of classroom(s) 
which accommodate various auditory and visual feedback devices. Visual 
feedback devices are placed so that the teacher can have access to the 
feedback information during teaching, with a minimum of classroom disruption 
These various feedback display devices are controlled directly by the com- 
puter. 
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Visual displays available in the Teaching Station (classroom) vary 
from the closed-circuit televised images currently employed, to various 
external visual devices which convey feedback information to the classroom 
by changing light pattems. 

Observation "Coding Station . 'ITie Observation-Coding Station provides 
the link between the events occurring in the classroom and the computer 
analysis of these events. Visual observation of classrooms may take place 
within the classroom itself, within an observation room adjoining the 
classroom (as in the present study) , or through a closed'-circuit television 
connection. There are three methods of data input employed by the CATTS 
system: a directly connected coding box device (used in the current project); 
a TOUCH-TONE telephone access system; and a portable recording unit for off- 
line data collection. The first two methods permit real-time data collec- 
tion for instantaneous feedback applications. The third method extends the 
system's data collection ability beyond the physical limitations of a direct 
computer connection into the field, and permits flexible collection of base- 
line data and delayed feedback observations ^ 

At present, the computer-connected coding device with which the observer 
enters his observations consists of a small hand-held box with 12 TOUCH-TONE 
pushbuttons. Currently, CATTS can accept data independently and simultaneously 
from 12 directly connected coding devices. This input time-sharing capability 
allows simultaneous data collection from various locations in order to ful- 
fill classroom observation requirements for independent projects. 

Analysis-Encoding Stati on. The Analysis-Encoding Station . contains a 
PDP-12 digital computer and associated computing hardware required for the 
on-line processing of pushbutton-coded data which/are gathered and transmitted 
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from Observation-Coding Stations. Tn addition to processing the incoming 1 
data, the computer system permits the control p£ various display devices | 
used in Teaching Stations and provides hard-copy printouts, storage, and \ 
transfer of data to remote computer systems for subsequent analysis. It isl 
at this station that an operator initiate^ .t-^ee computer program options 1 
available in CATTS. llie teleprinter consoJ . ::hrough software program con- \ 
trol, permits the operator to select spen- CATTS programs or options 1 
that satisfy various project data collecri:OT ^^d feedback requirements. ; 

1 

The selection of the mode and conter feedback to the teacher at the \ 

■ 1 

Teaching Station is initiated from the cons -e. If a CRT (Cathode Ray Tube) 
display is chosen as the method for feedback the operator determines the 
content of the display by assigning the incoming data, by code, to different 
computational functions for the computer to calculate and display as feed- 
back. The nature o£ the display is also selected from the console and 
allows feedback information to be presented either in alpha-numeric or 
' graphic form. . • 

Upon completion of an observation session, the raw data are stored in 
local computer magnetic tape units and become accessible for various data 
analysis applications. Analysis programs are available on both the PDP-12 
computer and Indiana University's central computer facility. For remote 
processinji,, the data are transferred by phone into permanent disk and tape 
storage files which are accessible by either batch or time-sharing analysis 
programs. 

The associated PDP-12 teleprinter, which also serves as the communica- j 

i 

tions link to the CATTS program, provides summary data printouts for inspec- ; 
tion throughout the data-gathering stage. The printouts provide such in- ■ 
formation as the time span of coded observations and the actual feedback 
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funcvi jns currently being displaye^^ l . the teacher. At the end of a real' 
time session, a selection of various data summary programs and printouts 
is available for delayed feedback applications. 



14 

i 

II- OBSERVATION SYSTEMS AND OBSERVER TRAINING 

The two observation systems used for feedback of teacher and piipi] 
behavior to trainees, via CATTS were the COG-STRAT and MAN-STRAT. Categories 
on the COG-STRAT include teacher and pupil co\gnitive styles of interaction, 
while the MAN-STRAT category is focused on pupil on- and off-task behavior and 
strategies of teacher management of pupil behavior (See Table 1 and also 
the Observer Training Manual, Appendix) . It is important to note 
that these observation systems are neither new nor original systems. Rather^ 
they were developed to meet teacher-trainee feedback needs for specific 
teaching skill development during their practicum experience- 

Prior to their practicum, trainees were enrolled in a course on 
systematic clasisroom observation. During this course they were exposed to 
a variety of observation systems and were required to individually develop 
systems based on their interests. In addition, all trainees and their 
instructor mutually agreed on a, number of teaching behaviors and corresponding 
pupil behaviors deemed important during classroom interaction with handicapped 
children* These behaviors were logically grouped into two domains, co? i ve ai 
affective management, and eventually became the COG-STRAT and MAN-STRAT systems 
Moreover, specific categories or groups of categories from existing observation 
systems were chosen whenever they coincided with trainees' needs- Systems 
from which categories were specifically taken and/or modified included the 
Individual Cognitive Demand Schedule (Lynch § Ames, 1972), the Indiana Be- 
havior Management System-II (Fink a Semmel, 1972) and the Flanders' Inter- 
action Analysis System (1970): A perusal of M irrors for Behavior 
' (Simon 5 Boyer, 1970) will also evidence other systems with categories similar 
to those selected (e-g.. Smith § Meux: Amidon)- 
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Table 1 
COC-STRAT and M^N-STRAT 
Observation System Categories 



MAN-STRAT 



Pupil Ca- 



Pupil Categories; 



1. 'dt iatCi I i K 

2. /-o^' Leve i ponse 

3. diigj -Levr ' '^'-sponse 

4. No3 Tespoir-.i-; 



1. On-Tai:k 
3- Off-Task 



Teacher Categories: Teacher Categories: 



D * 


Talk 


2. 


Reinfor ■:.:g 


6. 


Low- Le ve i Que s t i on 


4. 


Ignorii);., 




High-Leve:i!i Question 


5. 


Redirect! 


8. 


Positive iFeedback 


6. 


Commancii. 


9. 


Negative Feedback 


7- 


Focusirag 


10. 


Incorporat3-on/ Extension 


8. 


Norm RaferffiHiciij,^ 


11- 


No, Interaction 


9. 


Signal lin^i 


12. 


Interruption/Confusion/Management 


10- 


Humor 






11- 


Demeaning 






12. 


Unco dab le5 
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Also paramount in this selection process were considerations of simplicity, 
clarity, and objectivity of the observation systems, since trainees would be 
required eventually to learn the systems and code each other when teaching. 
For example, one reason that entire existing systems were not chosen was 
that they were too complex and/or contained categories irrelevant to trainee 
goals of teaching skill development. 

Once specific categories were selected and broadly d( fined , - y were 
given to a CATTS staff member who was highly experienced ir rnv opmem ^ 

jl'-.ervation systems and instructional programming of obsr i r g mat?'' ahr 

i.ritical attributes of categories were identified, and -ys- 'j*^^ were thr 

jHVv-...,ri:nental ly tested with a coder trainer system, ' '^lA, " -md 5ubi;feu|i[jen • ly 
reviS'^d. The operation of DITRMA will be described ;iere. in this r^p :.. 

Both aud \o and video tapes of classrooms and classroc a simulations were cov;ed 
on button boxes during developmental testing of the utility of the systems. 
End products of this activity were observer training manuals and rules for 
button -box coding (see Module 1) . 

Additional observer-training materials were then aeveloped for use with 
DITRMA: (1) Two scripts were written for practice coding with feedback in 
order to initially familiarize trainees with the categories and numbering 
schema before DITRI4A coding. (2) A number of video-taped, unambiguous, • 
isolated examples were then produced via simulation for initial training with 
DITRMA. Tliese examples were structured so thp^t they became increasingly 
more difficulty and they included more behaviors to code as the tape progressed. » 
By the end of the tape trainees were required to code 5 or 6 behaviors during 
a" 15- to 20-second interval. (3) A second video tape for each system consisted 
of a number of continuous 5-minute segments of classroom interaction. Segraehts 
were chosen from the TEL tape library on the basis of their coding difficulty 
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and representativeness of system categories. (-^ i inally-, crinierion .tapes 
were produced in order to test trainee corni^atCTir' ^ r, the obsereation systems- 
Each test was composed of two parts: One sect :i : sisted rrmdomly 
ordered, isolated, unajiibiguous sum xater- ; ' --i :las ;roor Iniaraction. 
•/<t>*proximately 10 or irore examplor: -1^1^^ w,.f?r- Uiivu 'li^ 

f -:^cond section of each video " ' i^r "^!L^ •'.tri? ^ simu . ..icn 

cf classroom inte'^'action whi^ «u;. tc^ l"- --^ i -dv . 

Foi lo^wiing production of the: trail. r j" ^lals, tn- wr^"- tc^?r. wr~ v 
.1 ':i.\mber af CITH personnel. In addition ::o t-:^r in back-up carders for tl\e 
iiracticum, the primary goals of thi:; testing wt : t.j estimate -raining tii.r' 
for scheduling purposes and to identify spc^ j f i ra-ning prob --ms . As . 
result of testing, modifications o( the train:^^^ prs-^c.-durf wer-e ade; but 
more important, a time shortage for schcdulin; of oUit^TV^r tr^^lk^ng became 
evident. It was obvious that it would be unxv .^conab to train 27 teacher 
trainees on two observation systems a.n two -:ee::s considering trninee class 
schedules and available time- In order to CTesrTvc enough time in the 
s.emester to allow each trainee to teach at Least 9 times and still maintain 
the design of the study, it was decicaed that: trainees could only competently 
learn one of the two systems in the allotted time period. COG-STRAT was 
chosen for the trainees since it appeared to be less difficult to learn on 
the basis of developmental testing- In order to utilize the other system, 
it was decided that CITH personnel would be trained to criterion on the 
MAN-STRAT, while trainees would also learn the MAN-STRAT (but not as coders) 
at the earliest possible time during regular class meetings. 



^A rationale for this testing procedure will bv: discussed in a later 
section. 
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Following is»*^i.opin6:ntal testing of training materials, final revisions 
were made, ^^lo -Mc..'vi,i, -:' i was prepared, llodule 1 es33£::nti;;.. y briefed students 
on what Wtti> ^^^i^^ .e^r^v.jed during obs^ -ver training. 't uncluded rationale 
for leamini^ ^;isf:r:*v^ j systems, a cie ^:iption of DETRMi/V, training schedules, 
a training .s -ilabus, j^: outside assi.rrr-Mt, the COG-STW.T Observer's Training 
Manual, Rules for Buti i Box Coding, a practice coding script with con- 

firmation. *dule 1 a o contained I/VN-STRAT Observer's Training Manual, 

Rules for Button Box C ding, and a ]) ^t-.ce coding script. (See Module I). 

Observer Trai^ ing. 

In the fi ^st st ^-e -ji- the practi::u-:., or the discrraination stage, trainees' 
learned the COG-STRA^' : ys-em. CITH pe-^sonnel also learmed the MAN-STRAT 
system at this time, /zavuisition of ooservation skillf- was facilitated by 
a computer-aided trainini;^ device called DITRMA. 

DJTRMA 

DITRMA is based upom a simple consensus coding principle whereby 
individual trainees' responses from two or more coding terminals are 
simultaneously compared by the computer, and whether or not there is agree- 
ment across observers' entries is instantaneously fed back to trainees. 
Through expanded application of this simple consensus principle, the DITRMA 
system can be used to teach Discrimination of relevant teacher-pupil behaviors, 
to TRain for reliability in recording those behaviors, and to MAintain that 
level of reliability (Semmel , 1975). 

DITRMA operates as follows: During coder-training sessions observers 
code video -taped examples of the observation system on button boxes. These 



19 



button boxes are similar in configuration to a TOUCH-TONE telephone and are 
linked by cable to a PDP-12 computer. Th»e computer acts as an impartial 
judge as trainees code with a small group of their peers. If they agree 
with each other on a certain event, the video tape keeps playing and the 
computer remembers that they agreed on this example. On the other hand, 
if there is disagreement us to how ;ui event should he coded, the computer 
automatically stops the video tape and "refuses'* to start the tape until 
the trainees, as a group, have reached consensus on what they saw. 

Physical configuration of DITRMA . For the CATTS practicum, the physical 
configuration of DITRMA consisted of two adjacent, interrelated stations: 
Training Station and lincoding/Feedback/Control Station. Included at the 
Training Station were a maximum of six on-line coding terminals, a closed- 
circuit video feedback monitor, and a video playback monitor. The Encoding/ 
Feedback/Control Station contained the computer, a Sony EV 300 video tape 
recorder (VTR) , the slave CRT scope and TV camera, and other associated 
hardware required for on-line processing of incoming data from the Training 
Station (see Figure 3) . 

The operation of DITRMA, Once an observer has initiated a ci^de on the 
button box, all remaining observers must enter heir codes within a given 
time interval. The length of this time interval can be varied, allowing the 
trainer to shape observer reaction time. If every entry is made before 
expiration of the set time interval, the computer compares the codes from 
each observer upon receiving the final data entry. If all observers agree, 
the computer enters this consensus data into its data storage buffer, and 
also sends an audio visual feedback signal to the trainees (see Figure 4), 
This sl»:pial not only serves as confirmation of agreement but also cues / 
trainees that they can enter new codes, if necessary. 
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If one or more observers disagree^ or if the set time interval expires 
before all entries are completed, several events transpire in a matter of 
microseconds : 

1. The computer instantly stops play of the video tape via a relays 
buffer to the video tape recorder; 

2. A disagreement is entered into the data buffer, and individual „ 
entries are stored in memory; and 

3. The exact nature of the disagreement is sent to trainees through a 
closed-circuit feedback monitor showing the data entry from each 
coding box (see Figure 4) . At this point the computer waits for 
the trainer and trainees to discuss the disagreement and come to a 
consensus, and it will store the amount of disciission time in memory. 
Upon reaching an agreement, the trainer enters the consensiis code 

on his master box in order to restart the video tape and continue 

the coding/feedback process. 
In addition to receiving instantaneoiis feedback on coding skills, each 
observer can receive delayed, hard-copy computer printouts via DITRMA which 
summarize agreements and disagreements during the training session. Thiis the 
trainer, as well as the trainee, can readily evaluate observer strengths and 
weaknesses and use this information in preparation for future training sessions 
Unfortunately, delayed DITRMA feedback was not available diiring observer 
training in the CATTS Practicum because of other higher CATTS coinputer- 
programming priorities. 

Observer Training in the CATTS Practicum 

An intricate training schedule was necessary in order to accommodate 
the 36 observer-trainees' available training-time schedules, trainer avail- 
ability, and free computer time. Training of different groins on both systems 
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of DITRM\ occupied as many as 12 hours per day of computer time, forcing TEL 
computer programmers to work nij^ht shifts during this two-week period. Tlie 
27 COG-STRAT trainees were divided into six groups of 4 or 5 members each 
and met for 1 to 1 1/2 hours once a da/, 4 days per week. liach group was 
rotated through available time slots and met approximately equally often with 
three experienced observatirn system trainers in order to guard against any 
specific trc. .ner biases that might develop. NIAN-STRAT training was accompl ishedv 
in a similar manner, although specific scheduling was worked around the less- 
flexible COG-STRAT trainee schedules. (See Table 1.2, Appendix I). 

During the first V<o days of COG-STRAT training, trainees familiarized 
themselves with the DITRMA system by coding trainer-read examples of class- 
room situations. The trainees had been instructed to construct these examples 
as a COG-STRAT familiarization exercise. Trainees also coded video-taped, 
isolated examples on DITRMA. The next two days were spent coding video-taped 
classroom simulations. Beginning with the second week of instruction, the 
trainees were given a criterion test. Upon analysis of this data the CATTS 
staff was unsatisfied with the level of observer agreement and training was 
continued for two additional days. At this time a second criterion measure 
was administered and all trainees reached a desired level of agreement. A 
similar training procedure was used with the nine MAN-STRAT observers. (See 
Table 1.5, Appendix I, and Module I). 

The total time needed for trainee acquisition of the COG-STRAT system 
varied from 8 to 10 hours,' while MAN-STRAT training took slightly longer. 
This includes time spent in training as well as time for taking criterion 
tests and retraining. 

Evaluation of DITRMA 

DITRMA was subjected to a thorough field test during the observer- 

'> 
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training phase o£ the present CATTS stud>'. Given the large number of 
trainees required to learn the obse^r^ation system and the short period of 
time allotted for training, the DITRMA field test was conducted under 'less 
than ideal conditions. Nonetheless, several conclusions seem warranted at 
this time. First, the ratio of five trainees to one trainer when viing 
DITRMA appeared to be not as efficient as anticipated. All trainers agreed 
post hoc that, although DITRMA can be used with six persons, a ratio of 
trainees to trainer of 3 to 1 is ideal for maximal exploitation of DITRMA. 
Second, it was concluded that the length of the training sessions on DITRMA 
should be limited to no more than 45 minutes to one hour at a time. Beyond 
this amount of time individiial trainee fatigue seemed to limit the utility 
of DITRMA for improving group discrimination skills. 

Despite the conditions under which DTTRMA was used, it seemed to sub- 
stantially reduce training time relative to traditional paper and p'=^ncil 
observer-training procedures. For example, total training time vsing the 
Indiana Behavior Management System- I I observer-training package (Fink 5 
Semmel, 1971) is typically 18-20 hours, while MAN-STRAT (a modified version 
of IBMS-II) training took approximately 10-12 hours with DITRMA. 

In addition, the job of the trainer was obviously less burdensome when- 
using DITRMA, since the computer monitored the trainees while coding and 
stored, retrieved, and analyzed observer-training data. With DITRMA the 
trainer's role is primarily reduced to that of a discussion leader during 
disagreement situations, and an ultimate authority for resolving disagree- 
ments or incorrect consensus among trainees. 

. Evaluation of Observer Competencies 

Observer competency on the COG-STRAT and MAN-STRAT systems was important 
for two reasons: I-irst, for trainees to accurately evaluate and change their 

on 
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teaching behavior using feedback from observers via CATTS, this feedback 
needed to be as precise as possible. Second, accurate coding by observers 
is a necessary^ though insufficient condition for experimental evaluation of 
different types of CATTS feedback- 

The issue of observer agreement and reliability of observational records 
has been discussed extensively elsewhere (see Frick 5 Semmel, 1974; Medley 
6 Mitzel, 1963; Medley 5 Norton, 1971; McGaw, Wardrop 5 Bunda, 1972), Among 
the major conclusions of these writers is that observers should be trained 
to nearly perfect agreement with a criterion or expert coder on unambiguous 
examples of behavioral categories before actual data collection. They should 
then be expected to agree on unambiguous events encountered in the field. 
But disagreement on ambiguous events observed in the field should also be 
expected, since such disagreement may help reflect- a more accurate representa- 
tion of what actually occurred. 

Since the amount of ambiguous events occurring in the field cannot be 
controlled, a measure of observer agreement in that situation is difficult 
to interpret. Rather, the best that can be done is to document that observers 
can accurately code unambiguous examples. This can be accomplished by 
showing observers a video tape containing only unambiguous isolated examples. 

This type of observer agreement measure is referred to as criterion- 
related agreement and was used to document that observers know the observa- 
tion system categories. 

In addition to criterion-related agreement, Frick 5 Semmel (1974) have 
recommended that measures of intra-observer agreement be obtained by showing 
a video tape twice to all observers in which conditions parallel those en- 
countered in the field. The purpose of an intra-observer agreement measure 
was to demonstrate the extent to which each observer cm code consistently 
under actual observational circumstances. 
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Both criterion-related and intracodev ineasures were suggested for 
observer maintenance checks as well. Ihe necessity of calculating agree- 
ment coefficients with the same type(s) of data (e.g., categor>^ frequencies, 
two-stage patterns, or scales) that are used in analysis of the actual data 
collected in the study was also emphasized (Frick § Semmel, 197 4). 

Thus, as previously stated, the CATTS staff produced a video tape which con 
tained approximately 10 or more isolated examples of each behavioral 
category for each system, and which also contained a 15-minute simulated 
classroom lesson to be coded twice in order to make decisions about observer 
competencies . 

It was decided on the basis of recommendations by Frick & Semmel (197 4) 
that simple percentage agreement _> .85 for each category would be required 
for the criterA on-related measure, and an overall proportion of agreement 
(Pq^) measure >_ .75 would also be demanded for the intracoder measure before 
observers would be allowed to begin classroom coding. 

Since data analysis using both category raw frequencies and category 
percent frequencies as units of analyses was intended, initial criterion- 
related agreement measures were calculated on both types of data. The 
formula used for calculating simple percentage agreement for each category 
for the criterion -related measure was: 

t^l ^o = ^1 or f^ , such that 0 <_ <^ 1.00 




where f^ = total frequency (or percent frequency) for the observer 

and ^2 " t:otal frequency (or percent frequency) for the expert (criterion). 

The formula used for determining an overall proportion of agreement 

(Flanders, 1967) for each observer on the intracoder agreement check was:. 
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i = l 



where = total frequency for first coding for the ith category 

and fj^ = total frequency for second coding for the ith category. 
2 

The reasons was used rather than the simple percentage agreement 
f 

measure [1] were that (a) Pq is not affected bv low frequencies of categories 

f 

as is P^; and (b) since observers tend to see more and code more during the 
second viewing, a proportion of agreement measure such as P seems more 
appropriate than a simple percentage agreement-. See Frick and Semmel (1974) . 

Since the video tape segment for the intracoder check contained ambiguous 
as well as unambiguous events, a check against a criterion was considered 
inappropriate and unnecessary because such a measure would be difficult to 
interpret (i.e.. What is an acceptable level of agreement ?)j and documenta-- 
tion that observers knew the- systems was already demonstrated via the 
criterion-related measure on solely unambiguous events. 

Moreover, TT^ (Pl'anders, 1967) was calculated for the intra-observer check. 
7T£ gives information about observer agreement beyond that indicated by P^ 
in that chance agreement and categorical distributions are taken into con- 
sideration. That is: 

[3] TT^ = " . ' 

1 - P 

e 

where P^ = proportion of agreement as in [2] 
f 

and P^ = chance agreement. 
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More specifically, 




where f. and f^ are the same as in [2]. Thus, tt^ is affected by the 

^1 '2 ^ 

number of categories in the system and the distribution of categorical 

frequencies whereas is not, since P^ tends to increase as the number of 

categories decreases and/or the distribution of categories becomes mor^ dis- 

proportional . 

On the other hand, ir^ tends to be positively biased when high positively 
correlated ratings occur (Frick 5 Semmel , 1974). For this reason the use of 
TT£ for a criterion-related measure for event recording systems such as COG-STPAT 
and MAN -ST RAT, when total category frequencies are the intended unit of analysis, 
can be misleading and was therefore considered inappropriate. 

However, the susceptibility of ir^ to correlated pairs of ratings, seems 

to be an advantage rather than a disadvantage of using it for an intra-obsejrver 

agreement measure, since observers often see more and code more during a 

second viewing. Thus, it was concluded that P and tt were the most appropriate 

0£ r 

measures to use for making decisions about adequacy of observer consistency. 
Procedure 

On the basis of the preceding rationale, several types of observer agree- 
ment checks were administered before and during the study. These will be 
referred to as initial and maintenance observer agreement checks, respectively, 
(See Table 1.4, Appendix 1). 
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For the initial checks on the two systems, both criterion-related and 
intra-ob server tests were administered. Observers independently coded the 
videotapes in groups (n per group ^6). Maintenance checks were similarly 
handled about one-third o£ the way through the study, except different video 
tapes were used. It should be noted that a maintenance intracoder measure 
was not possible with the COG-STRAT group due to severe scheduling problems 
and lack of trainee time to take such a test. 

In addition, "live'' maintenance checks were performed throughout the 
study on both systems in order to give the CATTS staff an indication of 
observer agreement with experts during actual trainee classroom lessons. 
This was also done for reasons beyond that of obtaining agreement estimates 
in situ . Since observers never kiiew exactly when an expert was going to 
double-code a given lesson, it was intended that observers would always 
anticipate such a possibility and come well-prepared to observe each time.' 
Moreover, expert double-coding was almost always done with observers whose 
skills were found through video and/or live maintenance testing to have 
deteriorated beyond prespecified levels. Double-coding continued with 
these problem coders until they obtained satisfactory agreemc^nt with 
expert. Until adequate agreement was attained, the experts' data were 
always used for CATTS feedback. 

Results: Initial Checks. 

Following observer training, both criterion-related and intracoder 
measures were obtained respectively for COG-STRAT and MAN-STRAT observers. 
When criterion-related checks were first administered, most observers did 
fairly well, but, with few exceptions, they failed to meet the preset standard 
of Pq ^ '^^ each category. Therefore, more training ensued before the 
second administration of the cri terion-relatedtests . Results of this 
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second test are given in Table l.S for -C0G-5TR.-M and Table 1.7 for MA\-STRAT 
(see -•\ppendix 1). Iz should be noted that each observer code consisted of 
two stages: Pupil Identification and Category Identification (See Rules 
for Button Box Coding, Module 1, Appendix). Therefore, in each agreement 
check both individual pupils and individual categories were considered. 

.As can be seen from Tables l.S and 1.6 in Appendix I, there is little 
difference on the average between simple percentage agreeiDcnt measures using 
raw frequencies and percent frequencies. Thus, in later maintenance criterion- 
related agreement measures, simple percentage agreement using percent fre- 
quencies was considered sufficient. 

With few exceptions individual COG-STRAT agreement scores in Table 1,5 
are greater than or equal to the preset standard of .85. If a trainee still 
had problems with one or two categories, he/she was quizzed by a trainer 
until the trainer was satisfied that the categories were completely under- 
stood. Since most trainees had some trouble with category #10 (Incorporation/ 
Extension), a special large group session was held in which the definition 
was further clarified and more examples were given for trainees to code:. 

As can be seevi from Table 1,6 in Appendix I, the MAN-STRAT initial 
criterion-related agreement measures were slightly lower on the average than 
the corresponding COG-STRAT measures. This was in part due to low frequencies 
of one category (* 12). Ir addition, three of the coders were housewives 
(#7, #8, #9) who had no prior experience with observation systems and who had 
more difficulty learning the NIAN-STRAT than did the remaining experienced CITH 
personnel. These three were yoked to trainers during the first two weeks of 
actual classroom coding until each trainer was satisfied that they were competent 
observers, Tlie trainers » data were always used for CATTS feedback during this 
time. 

In addition to initial criterion-related agreement, initial intra-observer 

n r- 



agreement iDeasures were obtained for both syster.s. .-^ can be seen from 

Tables 1.7 and l.S, ?^ and vere calculated. All COG-STR.^T coders ex- 

f 

ceeded the preset P ^ « requirer.ent for both puuil and categorv identifica- 

°f ' 

tion. In fact, almost all exceeded P > .85. IVitli one exception, NIAN-STRAT 

°f - 

obser\'ers likewise exceeded the preset standard. It can also be seen from 

Tables 3.7 and 3.8 that the rank-ordering of ''f*? is nln>ost identical to 

that of P 's. except that '^n^s are alwavs lower bv several hundredths. V 

°f ^f 
for pupil identification was almost alwavs higher than P for cr.tegory 

^f 

identification primarily because there were less than half as many pupils as 
there were categories — i.e., the probability of correctly identifying pupils 
by chance alone should have been higher than that for categor>' identification. 
Since an attempt was made for a fairly balanced distribution of categories on 
the intra-observer video tape segments, the •^-esult ng P 's were relatively 
small and barely exceeded expected probal ilirres c f char ce agreement alone 
(assuming every category was eniiall) likely to occur). ITius as Frick and 
Semmel (1974) have concluded, the correction for chance under these circum- 
stances should have had and did have li'ztle effect on thie ranuc -ordering of 

observers, based on P^ . . 

f 

Results: Video Maintenance Checks 

For M^N-STRAT coders, both criterion-related and intracoder agreement 
measures were obtained via video tape maintenance tests. The NIAN-STRAT intra- 
coder agreement check was administered about one month into the study, and 
its results are given in Table 1.9 (see Appendix I). Observers #4, #5 and ^7 

did not meet the P > . 75 standard. Observers .^4 and #5 were used only as 
o^ - 

substitute coders at that time in the study and had been coding very infrequently. 
Thus, it was decided that ^5 and #7 Would be yoked to trainers during actual 
coding sessions, while #4 dropped out of the project on her own volition. 
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The criterion-related check was administered about 2 1/2 weeks after 
the intra-observer test. Xs can seen from Table 1.10 in Appendix I, 
observers ^1 and -S did not take this test. Since observer ^1 v^as previously 
involved with the development of the test, he did not take it. Ol^scrver ^8 
was ill during this time period and could not take the test. She was later 
yoked to a trainer during actual classroom coding when she. recuperated. 

Some of the categories in Table 1.10 occurred with low frequency. In 
particular, categories ?2 aiid #4 and pupil occurred infrequently. Observers 
#3, #6, and ^9 misunderstood a special ground rule for coding category ^1 on 
the 90 isolated unambiguous segments. This ground rule was added only because 
the test was roken * to short segment : and was not used during actual coding. 
Thus, after briefly qu. " zing these Th:-!?e obi^ervers, it was evident that they 
obviously understood c:-.tfcgory but ^ ere confused abot?jt the special ground 
rule. 

It was also evident from Table 1.10 that several observers were experi- 
encing difficulties wnth a few specific categories. Tinis prompted a general 
review session in order to resolve any misunderstandings about category 
definitions that had arisen. 

Results of the COG-STRAT criterion-related observer agreement video 
maintenance check are given in Table 1.11 (see Appendix I). The majority 
of the 27 observers exceeded the P^' >_ .85 standard, although it was quite 
evident that about oile-third of the observers had developed misunderstandings 
on several specific categories- These observers were individually quizzed 
by a trainer on problem categories until he was satisfied that the categories 
were clearly .understood. Ntoreover, live maintenance checks were begun with 
each of these observers and continued until the trainers were satisfied with 
their coding performances. Trainers' data were always used for CXHS feed- 
back during these retraining sessions. 

ERIC ^ 
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Categories ^11 iSo Interaction) and -12 (Interruption/ Con fusion/Manage - 
!aent) were not reported in Table 1.11 because they occurred infrequently in 
the video tape test. This was of little concern, however, since none of 
the trainees had chosen these categories as teaching sl<ill development 
objectives . 
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III. PROJEtrr ORGA.\IZATION A.N'D I^f?LE^!r^TATrO^• 
The Teacher Education Laborator:/ Classrooins 

Training of teachers was conducted in laboratory classroons which were 
designed for the observation, collection and feedback of coded information 
on trainee performance. The classroor:5 were linked to the Teacher Education 
Laboratory (TEL) computer, and each room had observation stations (one-way 
vision windows) used by observers to collect coded observation systen data. 
There were three such classrooms established in which trainees conducted weekly 
half-hour language development lessons with small groups of pupils selected 
from a demonstration EMR class. 

Subjects , The subjects were 27 Indiana University undergraduate students, 
all special education majors enrolled in K490; Curriculum and Methods for 
Educable Mentally Retarded. The study and practicum were administered through 
weekly meetings of the K490 class. The 27 trainees met in these sessions 
for instructions, lectures, scheduling and communication or airing of problems 
encountered during the practicum. 

Grouping pupils for instruction . Each practicum student was assigned to 
teach the same group of children, once each week for the duration of the study. 
There were four to five children in each group and there were three such 
groups. These groups were drawn from a class of EMR children whose regular 
classroom was located in the TEL facility. The IQ range of the children in 
this class was 59 to 84. Age, sex and reading level data are shown in Table 
2. The instructional groups were designated x, y, z,and the assignment of 
children to groups was permanent for the duration of the study. 

The assignment of children to instructional groups was made by the class- 
room teacher in consultation with the Fiducational Director of the Developmental 
Training Center, Indiana University, whose responsibility included the TEL 

0,f 
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Table 1 

Assigr-nent of Pupils for Instruction 
Characteristics of Pupils in Each Group 



Pupil Group X 








rupll no. 


oex 


LA (.J an. ly/ij 


Reading Level 


01 


F 


7- 


Pre-primer 




M 


Q 


Primer 


03 


M 


9- 


Pre -primer 


04 


M 


11- 


Pre -primer 


Pupil Group Y 








rUpll .No. 


bex 


lA (Jan. 1973J 


Reading Level 




M 


"7 O 


Readiness 


02 


M 


7- 


Readiness 


03 


M 


/ — D 




04 


M 


7- 


Readiness 


Pupil Group Z 








Pupil No. 


Sex 


CA (Jan. 1973) 


Reading Level 


01 


M 


10- 


2.' 


02 


F 


9- 


1. 


03 


F 


10- 


2.2 


04 


F 


7-2 


1.8 


05 


M 


11- 


1.8 



*In some instances the CA's are approximate, but since the basis for group- 
ing was primarily reading level, the CA is presented for informational 
purposes only. 
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laboratory class. The grouping of pupils for instruction by CATTS trainees 
was based upon the similarity of instructional needs and the social maturi-fy 
of the pupil. 

Each group of pupils received a total of one hour of supplementary 
reading instruction by CATTS trainees per day. The same group of children 
was taught by one trainee for half an hour and by another trainee for half 
an hour. The three instructional groups (x, y, and z) were taught simultaneously 
in three different classrooms. See the weekly schedule of classes (Ap- 
pendix I) for the pattern of randomization of classrooms and day and hour of 
teaching. From this schedule, it can be seen that each group of children re- 
ceived instruction from nine different trainees during any one week of the 
study. Each trainee always taught the same group of children, that is, a 
trainee always taught either x, y, or z children. 

T rainee lesson planning . Self- instructional lesson-planning information 
was made available to the trainees in Module No. 2- The trainees 
were required to submit a written lesson plan for each lesson they taught. 
The instructional objectives for each lesson were given to the trainees two 
weeks in advance of the scheduled teaching time. Each trainee submitted a 
written lesson plan one week prior to teaching, for comment and evaluation. 
The lesson plans were read and evaluated by the staff member who wrote the 
objectives, and returned to the student several days in advance of the lesson. 
The module specified the items to be included in preparation of lesson plans. 
These items were prepared from guidelines to the course on instructional 'design 
for special education, which all trainees had taken the previous semester. 
Included in the checklist for writing lesson plans were suggestions on entry 
tests, task analysis, specification of objectives in performance terms, 
criterion tests for objectives and subobjectives, and teaching strategies. 
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In addition, several members of the staff, including the classroom 
teacher, were available for consultation to assist the trainees in analyzing 
and using the lesson objectives for preparing lesson plans, and adjusting 
tho level of instruction to the competence of the children in their groups. 

Curriculum . The curriculum selected to serve as the basis for che 
preparation of lesson objectives used by trainees for lesson planning was 
the series, Language Experiences in Reading (Van Allen and Van Allen, 1970), 
This series was adopted and modified specifically for the CATTS practicum to 
provide the basis for a supplementary reading program for the pupils in the 
laboratory class. The procedure for developing the objectives for each 
lesson included a task analysis of activities suggested in the Teachers 
Resource Book for Language Experiences in Reading , (Van Allen and Van Allen, 
1970). Contemporary English in the Elementary School , (Tiedt and Tiedt, 
1967) was also used as a resource, particularily those chapters with sugges- 
tions for supplementing and extending concepts and task. Instructional 
objectives that student teachers used for lesson plan preparation were stated 
in behavioral terms. Expected pupil terminal behaviors, which indicate the pupil 
attainment of a concept or ability to perform a task, were the basis for a 
given objective. 

Every attempt was made to assure the continuity of lesson objectives 
for each instructional group. To this end, modifications in the long-range 
curriculum plan were made as the need arose. This process was monitored by 
a staff member responsible for curriculum development and by the classroom 
teacher. Lesson objectives were designed for appropriateness to the instruc- 

tional level o"*: each group, and a specific objective was written for each fei^son. 

\ . 

A complete set of daily lesson objectives for each group is found in the 
trainee's Instructional Module. 
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Implementation Procedures: Teaching and Coding 

Table 3 indicates the main operations of the project, which were 
constrained by the multiple nature of the training and research objectives 
and also by temporal considerations. In place concurrently were the necessary 
computer software, training manuals for observer training, instructional modules 
for lesson-planning skills, instructional materials on the use of CATTS 
feedback and performance evaluation, and teaching and observation-coding 
schedules • 

The preservice trainees i'l the program were required to plan lessons, to 
teach one half hour per week, to evaluate their teaching performance based 
upon CATTS feedback, to develop individual goals for modification of teaching 
performance as indicated by.CATTS data, and to reliably code other trainees 
engaged in teaching. As described in Chapter II, a block of time was allocated 
for the training of the teachers tc become reliable coders of MAN-STRAT/ 
COG-STRAT Observation Systems. 

Baseline peri od; daily procedures. Following the coder-training phase 
of the program, each trainee received an individual schedule for teaching 
and coding, and also instructional objectives for preparing the next lesson- 
On the assigned day of teaching, the trainee arrived at the TEL and signed 
in. Following information posted in the laboratory, the trainee met the 
children in his/her group, and went to an assigned room. Trainees scheduled 
for coding followed posted information on specific observation station and 
button box assignments. These schedules were designed to counter-balance 
classroom and coder effects. 

Prior to teaching, most trainees began by setting rules, establishing 
routines or raoti\*ation for the lesson. As soon as teaching started, coders 
began recording categories of behavior on the coding boxes. The coding 



. I Table 3 . - ^ 

CATTS Project Operations and Time Line for Development and Implementation 



OPERATION 


, DECEMBER 


JANUA 


«i 


FEBRUARY 


MARCH 


APRIL 


MAY 


JUNE ■ 


Hardware/Soft- 
ware Development 




DITRMA 


^eady 


CA'ITS Feed- 

DaCN KcdCiy 


CATTS Feedback 


CATTS Feedback 


■CATTS Data 
Transmission 
for Analysis 




Obsemtion 
System Develop- 
ment 5 Discrimi- 
nation Training 


Selection of 
COG-STRAT HAN- 
STRAT Cate- 
gories. Develop 
ment of Train- 
ing Manuals 6 
Materials 


Observe 
Trainin 
Criter^ 
■(Module 

-) 


[ 

P to 
on, 

V 


Observation 
Coding 


Conduct Reli- 
ability-^ — 

Checks 


. ) 

^ 














Curriculum 5 
Lesson Planning 


Group Pupils. 
Select Curric- 
ulum. 


Develop Lesson 
Objectjives 5 
Sequence 
(Module 2: 
Lesson Plan- 
ning). 


Staff Support 
System 

Consul' 
Select: 


.ation on Lesso 
on of Perforraa 


. , ) 

1 Planning, 
)ce UDjectives. 






Teaching 

1. Base line Obser- 
vation Period 

2. Teaching with 
Feedback 

(Generation 
Phase). 




Schedule 
Teachers and 
Observers 


Start Base- , 

line 

-Selection of 
-(Module 3 5 

Graphing § In' 


leacning uojec 
4) Begin 
CATTS F.B.(T 


cives 

reatment) ^ 




1 


;erpreting Beha 


^lors . 


Evaluation 
ERIC 






Formative 
Evaluation of 
Baseline 
Teaching, 
CATTS Post' 
Teaching F.B. 
Questionnaire. 
Trainee Evalu 


Trainee 
Evaluation of 
Practicum 
(Group Meet- 
ings] 


_ ^ 


Final Trainee 
Evaluation of 
Practicum 


1 

:Data;Analysi5 

•i: 

\ 


ition of Teachi 


ig Performance 

> 


(Graphing)— 
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continued for thirty minutes,' after which the computer automatically shut 
off the coding terminals. Even in those instances where teachers continued 
the lesson, only the first thirty minutes of coded data was taken. In those 
cases where the teacher trainee terminated the lesson before 30 minutes, the 
coders ceased recording when it was clear that the lesson was over. One 
half hour after teaching, the trainee took his place at the observation 
station and coded another trainee's lesson on the COG-STRAT System. In 
this manner, trainees reversed tlie coder-teacher roles. 

Depending on a preset schedule, whi ch was in turn based upon the re- 
search design, each trainee taught at least three and up to six times, with- 
out receiving any feedback on teaching. This period of teaching without 
feedback resulted in the establishment of a* baseline of individual teaching 
behaviors. All baseline data transmitted by the coders on the COG-STRAT 
and MAN-STRAT systems were retrieved, printed out, and copies were distributed 
to the trainees before feedback treatment began. 

Feedback phase . Prior to the first teaching session with CATTS feed- 
back, each tTp'^'' rec- ived a copy of the COG-STRAT and MAN-STRAT printout 
for each of t baseline lessons taught. Immediately after the trainees* 
final baseline teaching session, CATTS staff members instructed the trainee 
in the interpretation of the printout information. This was supplemented 
with Module No. 3, which provided the same information in a self-instruc- 
tional format. 

Graphing baseline behavior . The trainees were instructed in procedures 
for using thr^ Printout information to prepare graphs showing the percent 
frequency of occurrence of behaviors for each of the observation system cate- 
gories. They were also assigned the task of graphing all categories in the 



baseline period and required to evaluate their teaching performance on .the 
basis o£ their graphs and to also select three behavioral categories that 
they judged as most important to their teaching. They were then instructed 
to consider their baseline performance on the three preferred categories and 
choose one category from among the three that had shown a relatively stable 
pattern from lesson to lesson during the baseline period. This choice be- 
came, in most cases, the category that was targeted for improvement in sub- 
sequent lessons. The trainee also had to indicate whether s/he wanted to 
accelerate or decelerate the percent frequency of occurrence of the target 
behavioral category in subsequent lessons. The graphing assignment and 
choice of teaching behavior category was completed prior to the first lesson 
taught with feedback. 

All trainees received the graphing -printout instructional module (No. 3) 
after their final baseline teaching sessions. This was used to supplement 
the information given to the trainees in small group instructional sessions 
held immediately after the last baseline lesson. These sessions were held 
so that each trainee would know how to interpret the baseline data and 
make an informed decision on which category of behavior to choose for 
improvement. 

VTR feedback . Half of the trainees received instruction on interpreting 
the video scope feedback. Module 4 (Appendix) 'was developed for self-instruc- 
tion in video scope interpretation. The trainees assigned to receive video 
scope feedback were also given video scope instruction by a CATTS staff member 
in the TEL, which included viewing a simulated data display. ' 

Instruction in interpreting and using CATTS data, either from printout 
or from printout and video display, always occurred between the trainees' 
last baseline lesson and first lesson with feedback. 
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Durinp the feedback stage, the weekly lessons continued as during base- 
line and, in addition, all trainees received printouts on their latest lesson 
within two hours after teaching',. The group receiving video feedback always 
had in their classroom a video monitor which showed the cumulative percent 
frequency of occurrence of the targeted behavioral category that the trainee 
had chosen for teaching behavior change. In addition, a video monitor was 
placed in the rooms of teachers who received printout .feedback only. In 
these rooms, the VTR showed the same graph as the VTR feedback group had 
but with the baseline arbitrarily set at about 20^6 and a continuous poi^t 
of light flashing on the x axis at short time intervals. Tlie cumulative 
points constituted a straight line on the ordinate over the 30-minute period. 
Figure S shows VTR display for scope feedback and printout-only, feedback groups* 
Focusi ng the trainees* a ttent ion on the tar get ed behavior a l category . ' Sev- 
eral steps were taken to assure that the trainees would attend to the category 
chosen for feedback and behavioral change. Tliat is, measures were taken to 
focus the trainees' attention to the skill development task that was imposed 
as the primary goal of the practicum experience. Tfie printout-only group 
was required, to graph the percent frequency for the targeted category after 
the printout was received. This data was recorded in a cumulative, individual 
graph for the category and. included the trainees* baseline pc^cr'ntages In 
addition, all trainees kept a running record of the percent lequenc. in 
category for each successive lesson by entering the information for the 
previous 'lesson on a graph in the laboratory. The trainee did this immediately 
before joining the pupils and beginning to teach. This, in effect, was a 
redundant activity since the trainee had already entered the percent frequency 
information in his own record book after teaching, and then repeated the same 
entry in another book^ immediately before entering the crassroom to t'each 
another lesson. 





Figure 5. VTR display for scope feedb,ack group and printout group. 
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V The video feedback group was shown a post-teaching questionnaire during 
the scope training period and informed that it would be administered after 
each lesson they taught. This information was reiterated in Module 4, CATTS 
Video Feedback. It was expected that the questionnaire would serve to re- 
inforce the trainees attention to the scope during teaching. Among items in 
the questionnaire where attention to scope feedback was requisite, was question 
No. 2, in which the trainee was required to estimate the number of times he 
looked at the monitor. Question No. 3 required the trainee to reconstruct 
the video feedback display for the entire 30-minute lesson (See questionnaire 
in Module 4) . 

The number of times a trainee looked at the feedback monitor was also 
independently tracked by a third observer in the coding booth, as a cross- 
validation measure. 

Scheduling of Teachers and Observers 

Conditions for the preparation of the schedule for the CATTS teachi , 
sessions were determined by the constraints of the statistical design of the 
study and required appropriation of the following variables: 

1. Assignment of trainees to pupil groups. 

2. Assignment of trainees to feedback conditions. 
'3. Rotation of room assignments. 

4. ^ Assignments of trainees to day of teaching. 

Assignment of train ees to pupil groups . The trainees were assigned to pupil 
groups (x, y or z) on a. random basis. Table 4 shows the three pupil groups 
and the identification number of the trainees assigned to teach each group. 

Assignment of trainees to feedback conditions . There were three con- 
trolled conditions under which the trainees taught. These were: (1) the 
baseline, (2) the printout feedback only , and (3) the scope and printout 



ERIC 
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Table 4 

Assignment p£ Trainees to Pupil Groups 



Pupil Group 



Teacher Trainee ID Number 



\ 



ERIC 



14 


05 


20 


01 


26 


02 


06 


09 


12 


24 


03 


21 


10 


18 


22 


04 


16 


23 


19 


07 


17 


15 


11 


28 


1.'. 


08 


25 
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feedback condition. In the baseline condition, all trainees taught the lessons 
they had prepared without receiving feedback of any kind. The number of 
lessons taught without feedback varied from trainee to trainee. The baseUne 
teaching always included at least the first three lessons and varied up to' 
the first six lessons taught. 

The second controlled condition was teaching after the trainee received 
computer printout information about his/her baseline teaching performance and a 
set of instructions (Module No. 3) on how to read the computer print- 
out. At this stage the trainees were also required to evaluate their base- 
line teaching and to select by listing in order of personal preference three of 
the observation system categories they wished to work on for improving their 
teaching performance. The instructions for choosing a behavioral category 
for teaching performance improvement was included in Module No. 3 . 
Within a week of receiving baseline information, the trainees were required 
to select a single behavioral category fox subsequent improvement. The final 
selection was arrived at in consultation with a staff member. All of those 
activities were carried out after the baseline period and before the trainee 
taught with feedback. 

In addition to these procedures, the trainees were randomly assigned to 
feedback conditions. Half the trainees were assigned to a printout-only 
feedback group, and half to a scope and printout feedback group. The scope 
feedback group was presented with a self-instructional module (Module No. 4) 
and also received instructions from a staff member on how to use CAITS video scope 
feedback. This step was performed prior to the trainees teaching with feed- 
back. Table 5 illustrates the variable baseline and the two types of feed- 
back conditions. 

er|c 
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Table 5 
Schedule of Baseline, » 
CATTS Instruction and Feedback Conditions 



Week 


1 


2 


3 


4 


5 


6 


*7 
/ 


8 


9 


10 


Design 






















1 


A 


A 


AX 


B 


B 


B 


B 


B 


B 


B 


2 


A 


A 


AXY 


C 


C 


C 


C 


C 


C 


C 


3 


A 


A 


A 


AX 


B 


B 


B 


B 


B 


B 


4 


A 


A 


A 


AXY 


C 


C 


C 


C 


C 


c • 


S 


A 


A 


A 


A 


AX 


B 


B 


B 


B 


B 


6 


A 


A 


A 


A 


AXY 


C 


C 


C 


C 


C 


7 


A 


A 


A 


A 


A 


AX 


B 


B 


B 


B 


8 


A 


A 


A 


A 


A 


AXY 


C 


C 


C 


C 



Conditions : 

A = Baseline, no feedback 

B = Printout feedback only 

C = Video scope and printout feedback 

X = Instruction on CATTS printout and graphing 

Y = Instruction on video scope use 
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Assignment of trainees to baseline and feedback conditions was random. 
Table 6 shows assignment of trainees to feedback conditions and the inter- 
vention of instruction on the use of CATrS printout and graphing, as well as 
instruction on the use of the video scope feedback. 

Coder scheduling . The scheduling of COG-STRAT coders, all of whom were 
trainees, depended upon the trainees' teaching assignment. For example, if 
a trainee taught during the first half hour, she/he coded during the second half 
hour. The t'^iinee who coded the first half hour, taught during the second 
half hour. The MAN-STRAT coders were either staff members or trained MAN- 
STRAT coders employed for the project. Assignment of trainees to day of 
teaching was random. Coders were also randomly assigned to teachers such 
that each teacher was observed by a different coder in her/his group each time. 
Teaching room assignment was likewise rotated between three rooms in the TEL 
facility. An assignment schedule was prepared in advance of actual teaching 
and distributed to"* each trainee and coder as well as to the staff, Thv. 
master schedule for the entire project was posted in the TEL so that the day, 
hour and conditions (F.B., no F.B.) were known in advance to coders, trainees 
and staff. From time to time, adjustments' in schedules were neces. ary due to 
absences. These -were made up under conditions which matched the original 
assignment as closely as possible. 

Control procedures . The teaching and coding schedule was prepared in 
advance o^ the first teaching session, and a few adjustments were made to 
accommodate some conflicts in assignment with other personal committments. 
Once the schedulye was final ized",a master schedule was posted in the TEL and 
copies were given to each trainee. All persons participating in a teaching 
session signed in before beginning to teach or code. 



5:; 
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Table 6 

Assignment of Trainees to Feedback Conditions 



Feedback Condition 
Week 1 2 3 4 5 6 7 8 9. 10 

Trainee 
lb Number 



m 


A 
t\ 


A 


A 


A 


AY 
AA 


o 


n 
D 


D 

13 


D 
D 


D 
O 




A 
/\ 


A 


A 


A' 

A -V 


D 
D 


D 
D 


D 

D 


D 
D 


D 
D 


D 
D 




A 


A 


A 


A 


AYY 

r\A I 


r 


r 


r 


r 


r 

L( 


HA 


A 
t\ 


A 


A 


A 
A 


AY 
AA 


D 
13 


D 
D 


D 
D 


O 


D 
D 




A 


A 


A 


A 

A 


AY 

rVA 


D 
D 


D 
D 


D 
D 


D 
D 


D 
D 




A 


A 
r\ 


A 


A 


A 


AYY 

/AA I 


r 


r 


r 


r 


u / 


A 


A 


A 


AYY 

/\A I 


r 


r 


r 


r* 


r 


L» 


Uo 


A 


A 


A 


A 


A 
A 


AY 
AA 


D 
D 


D 
D 


R 


D 
D 


no 


A 


A 


AYY 


r 




r 


r 


r 


r 


r 


10 


A 


A 


AXY 


C 


c 


c 


C 


C 


C 


c 


11 


A 


A 


A 


A 


A 


AXY 


C 


C 


C 


C 


12 


A 


A 


A 


A 


A 


AXY 


c 


c 


c 


c 


13 


A 


A 


AX 


B 


B 


B 


B 


B " 


B 


B 


14 


A 


A 


A 


A 


A 


AX 


B 


B 


B 


B 


15 


A " 


A 


A 


A 


AXY 


C 


C 


C 


C 


C 


16 


A 


A 


A 


A 


AXY 


C 


C 


C 


C 


C 


17 


A 


A 


A 


A 


AXY 


C 


c ■ 


C 


C 


c 


18 


A 


A 


A 


AX 


B 


B 


B 


B 


B " 


B 


19 


A 


A 


A 


A 


A' 


K{ 


B 


B 


B 


B 


20 


A 


A 


AXY 


C 


C. 


c 


C 


C 


C 


C 


21 


A 


A 


A 


A 


AXY 


C 


C 


C 


C 


C 


22 


A 


A 


AX 


B 


B 


B 


B 


B 


B 


B 


23 


A 


A 


A 


A 


A 


AX 


B 


B 


B 


B 


24 


A 


A 


A 


AXY 


C 


C 


C 


C 


C 


C 


25 


A 


A 


A 


A 


A 


AXY 


C 


C 


C 


C 


26 


A 


A 


AX 


B 


B 


B 


B 


B 


B 


B 


27 


A 


A 


A 


A 














28 


A 


A 


A 


V A 


A 


AX 


B 


B 


B 


B 



Conditions : 

A = Baseline, no feedback 

B = Printout feedback only 

C = Video scope ai^d printout feedback 

X = Instruction on CATTS printout and graphing 

Y = Instruction on video scope use 
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The Study of the Effects of CATTS Feedback on Trainee Behavior s 

The empirical questions addressed in the present study were as follows: 
1} \Vhat is the effect of providing CATTS instant (video scope) fet^dback 
displaying trainees' performance in a target behavioral category, and also 
providing post- teaching printout summaries of coded teaching behaviors, 
upon the rate of generation of targeted behavior as compared to the base 
rate of the behavior? 2) ^liat is the effect of receiving only post-teaching 
feedback (printout summaries) upon the rate of generation of the targeted 
behavior, compared to the base rate of behavior? 3) WTiat is the relative 
effectiveness of CATTS instant and post -teaching printout feedback, compared 
to post-teaching printout only? 4) l\1iat are the effects of targeting and 
feedback of behaviors on the rate of generation, compared to rates generated 
in categories of behavior not targeted for feedback? 



IV. RESULTS AND DISCUSSION 

The statistical design lised to analyze the CATTS project data was a 
repeated measures ANOVA design with two between-block factors --Group (G) ' 
and Treatments (T)--and one within-block factor, Periods (P), Groups refer 

-to the three reading groups [X, Y, Z) to which, the classroom children were 
assigned. Eight teacher trainees were then randomly assigned to each ^ 
readrng gr^^ 24 subjects overall. Treatments refer to the 

two types of feedback treatments randomly arsigned to each half of the 
subjects within each groijp; Tj refers to the group of subjects who received 

'the hard-copy printout only, and T2 refers to the .group who received the 
instantaneous scope display in addition to the printout. Periods refer to 
the baseline and treatment periods of the study, P^ refers to the average 
of the baseline teaching trials and P^ refers to the average of the treat- 
ment teaching trials for each subject. This brings the total number of 
observations for the analysis to 48, The original design contained a fourth 
factor, different variations of baseline and treatment trial combinations, 
but due to inconsistent teaching schedules and some missing observations, 
this baseline/treatment factor was not fully completed, A preliminary 
analysis, including an incomplete version of this fourth factor, showed no 
difference at all between the various baseline/treatment combinations. The 
decision was then made to collapse each subject's trials within the various 
baseline and treatment periods and calculate one average score per period. 
The rate of responding on the trainees' chosen feedback category was 
selected for the dependent measure employed in the analyses , This criterion 
measure wa§ calculated by dividing the frequency of occ^irrence of the feed- 
back category during any one session by the time of that session. This 
measure was selected because it was considered more stable than frequency 
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or percentage of frequency, due to the varying length of some teaching 
sessions. The rate measure also provides a method of combining under one 
measure the two orthogonal components of frequency and time. 

Table 7 contains the analysis of variance source tables for the 
feedback category as well as the second and.Xhird choice categories. F 
ratios for the feedback category indicate that there were no significant 
differences in the Groi^ (G) and Treatment (T) m?,in effects, while there was 
a significant finding in the Period (P) main effect (p<.01). An examination 
of the means in Table 8 and Figure 6 reveal tfiat feedback trials were 
significantly higher than baseline trials. In addition , the Treatment by 
Period interaction (TP) was also significant (p<.05). All other inter- 
actions were statistically nonsignificant. 

Due to the significant TP interaction, a qxialif ication of the main 
effect of T and P is required. Figure 6 illustrates the TP interaction , 
plot of the mean scores from Table 8 on the criterion feedback category. 
A simple main effects analysis (Kirk, 1968) was performed on T and P in 
order to qualify these main effects. The results of the simple effects 
analysis are also indicated, in Table 8 . 

As expected, the qualification of the Treatment effect (T) indicated 
that the scope and printout feedback condition (T2) and the print outr-only 
feedback condition (T^) were not significantly different during baseline 
trials (P^), but did reach significance (p<.05) during the treatment trials 
(P2^ ' Period effect (P) is qualified by showing that the printout— only 

treatment group (Tj) did not significantly increase their mean rate of 
responding between baseline (P^) and treatment (P2) trials, while the scope 
display -printout treatment group (T^) increased significantly (p<.01) between 
baseline and treatment trials. 



Table 1 

Analysis of Variance Table for Feedback and Second and Third Choice Categories 



SOURCE 



FEEDBACK CATE'GOAY 
SS df MS F 



Between Subjects IS0S2 23 
Groups (G) 
Trcatr-cnts (T) 

Btvn T at Pj 

3:wn T at P2, 

I'-'ithin cell 

GT 

S w/ groups S(GT) _ 



.0711 


2 


.0356 


(1 


.172S 


1 


.1728 


2.79 


.0000 


1 


.0009 


(1 


.3435 


1 




7.15* 


.57% 


12 


.0^33 






2 


.0753 


1.19 


1.1167 


18 


.0620 





Kithin Subjects 
Periods (P) 
Btwn ? at Tj 
Btwn P at T2 
CP- 
TP 
CTP 

j6 w/ groups SP[Gri 
Total 



1.6212 


24 






.7301 


1 


.7301 


2i.r 


.0%2 


1 


.0%2 


2.78 


.8066 


I 


.8066 


23.31** 


.0852 


2 


.0426 


1.23 


.1723 


1 .1723 


4,99* 


.0096 


2 


.0048 


(1 


,6235 




_.0346^ 




3.1294 
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SECOND CHOICE CATEGORY WIM CHOICE CATEGORY 
SS df ' MS F SS df HS F 



7.9394 23 
.7119 2 .3559 <1 
.2806 1 .28C6 <1 



6.2206 

1,0766 2 .5383 2.34 
,1055 1 .1055 a 



.2739 2 .1369 <1 .8953 2 M% 1.95 
6.6730 18 .3707 4.1452 18 .2303 



5104 24 ' '^^^^ „ 

:0054 1 .0054 (\ .0105 1 .0105 <1 



.0024 2 .0012 a .0065 . ( 

.0609 1 .0609 2.67 .0137 . .02 

.0306 2 .0153 <1 .0014 2 . d 

.4111 18 .0228 .240iiL.iE_ 



8.4498 47 



6.4931 47 



*P<.05 
♦♦P(.01 
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Table 8 

Mean Kate Performance od Categories and Treatment Phases 

p2 INCREASE FROM 

BASELINE TREATMENT BASELINE 



SELECTED Printout only T -1350 .2716 .1267 

FEEDBACK Scope 5 printout T^ .1350 .5017 .3667 

CATEGORY XT^ ^ ^ .1350 -3867 .2467 



SECOND Printout only T .2675 .2175 -.0500 

CHOICE Scope 5 printout T^ .3492 .4417 .0925 

XT 5 T .3083 .3296 .0213 



THIRD Printout only T .2908 .3542 .0634 

CHOICE Scope 5 printout T^ .2308 .2267 -.C041 

XT 5 T .2608 .2904 ,0296 
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In order to further support the simple main effects interpretation of 
the significant TP interaction, a Tukey post hoc analysis was performed on 
the means which were plotted in Figure 6. The saine results were fou^- . as 
reported above with the same alpha level of significance. 

As indicated previously, prior to the trainees' selection of their 
criterion feedback category, they initially chose three cognitive teaching 
behaviors that they specified they would like to work on in the classroom* 
From this list of three., the final feedback category was selected* As a 
contrast to *he selected feedback category, identical analyses were per- 
formed on the second an^. third choice behaviors that were not included in 
the concentrated CATTS feedback program* These second and third choice 
behaviors were not displayed on the scope instantaneous feedback display, 
but were included on the printout given to both treatment groups together 
with all of the other behavior categories coded in the classroom* 

An examination of Table 7, Variance Components for Second and Third 
Choice Categories, revealed no significant main effect differences or in- 
teractions present in the data* Hence, only the criterion feedback category 
selected from the original chosen three categories was found to improve 
significantly from baseline to treatment* Figure 7 (a § b) graphically 
illustrates the mean rate increase from baseline and performance differences 
from baseline between the three feedback choice categories* Figure 8 
further illustrates the differential effect of the two treatment feedback 
conditions, T^ and T^, across all three categories* 

F max tests (Kirk, 1968) were performed on the error terms for all three 
categories found in Table 7 in order to test whether or not the assumption 
of homogeneity of the partitioned parts of the within-cell variation is 
tenable* F max ratios indicated that both the within- subjects and between- 
subjects variances error terms for the feedback category were found to be 
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made up from homogeneous within-cell conponent*^* The within-subj ects error 
term for the second choice category was also found to be homogeneous. The 
F laax ratios for the remaining error terns rejected the hor.ogeneity 
assumption. The app.\ icati-jn of this test qualifies the interpretation of 
the second and third choice categories, but lends support to the results 
attributed by CATTS to the chosen feedback category* 

Figures 9 (a, b, c, § d) are plots of the groups of trainees who were 
included in the initial four t\^es of baseline/treatment combinations. Most 
of the missing obsen'ations for the trainees occurred during the eighth, 
.ninth, anci tenth treatment trials. As a result, the projected points towards 
the end of the teaching periods reflect considerably reduced n's, and there- 
fore are unstable* Missing observations also occurred during the initial 
baseline trials but were considerably fewer in number, so the stability of 
the baseline trials is fairly consistent with n's of 6 and 7. 

Although 25 subjects were used in the analysis, there was data available 
from 27 trainees. Three of the trainees had identical baseline/treatment 
patterns to three other trainees, and the decision was made to combine those 
three pairs into three separ mean scores and provide equal cell sizes 
for analysis* 

'-The results of the study revealed that all trainees in both treatment 
conditions significantly increased their rate of criterion performance by a 
ratio of 2.8:1 over the baseline^ as a function CATTS feedback. There 
•■•were no differences between the two CATTS feedback groups during baseline 
on their individually chosen criterion measure* However, during treatment 
trainees in the CA'iTS scope and printout condition increased their criterion 
rate of responding to a significantly greater degree (2:1 ratio) than did 
trainees in the delayed ;irintout- only condition* 



.2.00 

1.80 

1.60 

(J 1.40 
c: ■ 

' 1 1.20 

5*? 1.00 

0 

U. 0 

UJ 

a. 



)! .80 
,60 
40 

.20 

,00 









Ttiferboseline-ieyenlrtotmept 




observoiions ^roup N=6 








Trjolment 

, 1 1 1 1 .L^l_ 



J 3 4 5 6 ? 8 9 
TEACHING PERIODS' 



2,00 
'i';60 

1,60 



UJ 140 
a .i 1,20 

s:,80 

a." .60 



,40 

,20 
,00 



Foyfbo5eliiie-5iii Ifeolfiniit 
obsefvolioos gfoyp W'l 




2 1 4 5 6 ?? 9 
TEACHING PERIODS 



Figure 9a. Performance rate curve for the 
3/7 group. 



Figure 9b. Performance rate curve for the 
4/6 group. 
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V. SUMMARY AND CONCLUSIONS 
This project attempted to demonstrate the relative effectiveness of 
CATTS instantaneous and delayed fee^^back in generating specific teacher 
behaviors in a preservice teaching environment- The project built on previous 
developmental work at Indiana University involving the CATTS system* Specif- 
ically, the purpose of the project was to demonstrate the effect of CATTS 
immediate visual and delayed (printout) feedback on increasing various cognitiv 
and management behaviors of teacher-trainees in a controlled laboratory class- 
room setting- The investigation was carried out in conjunction with the 
practicum phase of a junior level course in special education at Indiana 
University. 

There were 27 majors in special education who participated in the study- 
In all, the study entailed three major phases: first, discrimination training 
in which trainees learned an observn*^lon system; second, collection of a 
baseline of trainees* cognitive demands and behavioral control strategies in 
the classroom; and third, measurement of trainees' cognitive demands and 
behavioral control strategies under CATTS ias.tantaneous and/or delayed feed- 
back conditions. 

In the first phase of the project trainees learned two category obser- 
vation systems entitled COG-STRAT and MAN-STRAT. Categories on the COG-STRAT 
included teacher and pupil cognitive styles of interaction, while MAN-STRAT 
categories focused on pupil on- and off-task behavior and strategies of 
teaching management of pupil behavior. The behaviors chosen to be included 
in these systems were those which were deemed important during classroom 
interaction with handicapped children- All trainees were trained as coders 



on the COG-STRAT system. CITll personnel were trained as coders on the MAN- 
STRAT system. Aaiuisition of obscrvation-coding skills was facilitated hy 
a computer-aided training device called UITiytA. Tlie total time for trainee 
acquisition of the COG-STRAT system varied from 8 to 10 hours, while MAN- 
STRAT training took slightly longer. 

Following observer training, both criterion-related and intracoder 
measures were obtained respectively for COG-STRAT and MAN-STRAT observers 
via video-taped criterion tests. All trainees achieved initial criterionr 
related agreement scores >_^,85. In addition to initial criterion-related 
agreement, initial intra- observer agreement measures were obtained for both 
systems. All trainees exceeded the preset standard of .75 for both pupil 
and category identification. In fact, almost all exceeded a .85 standard. 
Maintenance checks for criterion-related agreement were similarly performed 
about one-third of the way through the study, except different video tapes 
were used. In addition, "live*' maintenance checks were performed throughout 
the study on both observation systems in order to monitor the observers' 
agreement with expert coders during actual trainee classroom lessons. The 
majority of :\e trainees exceeded the .35 standard on the maintenance check. 

In each of the two teaching phases (i.e.. Baseline and Feedback), 
trainees taught language arts lessons they had prepared. Lesson objectives 
that trainees used for lesson plan preparation were stated in behavioral 
terms. Expected pupil terminal behaviors, indicating the pupils attainment 
of a concept or ability to perform a tas':, were the basis for a gi\''^n objec- 
tive. The trainees were required to submit a written lesson plan for each 
lesson they taught one week prior to teaching, for comment and evaluation. 
Each practicum student was randomly assigned to teach the same group of 
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children, once each week for the duration of the study. There were four to 
five children in each ^roup and there were three such groups, 'fliese groups 
were drawn from a class of EMR children whose regular classroom was located 
in the Teacher liducation Laboratory (TEL) at CITII. The I.Q. of the children 
in this class ranged from 59 to 84, and they were grouped for instruction 
according to the similarity of instructional needs and social maturity, fiach 
group of pupils received a total of one hour of supplementary reading instruc- 
tion by CATTS trainees each day. The same group of children was taught by 
one trainee for half an hour and then by another trainee for the next half 
hour. 

In addition, each trainee coded another trainee in his/her group at least 
once a week on the COG-STRAT system. The scheduling of COG-STRAT coders 
depended upon the trainees' teaching assignment. For example, if a trainee 
taught during the first half hour, s/he would code during the second half 
hour and vice versa. Coders were randomly assigned to teachers and sched- 
uled so that each teacher was observed by a different coder on successive 
lessons, and assignment of trainees to day of teaching was also random. 
Assignment of trainees to teaching room rotatJ=;d between the three classrooms 
in the TEL facility. Trainees observed and coded classroom interaction on 
button boxes which were linked to a PDP-12 computer located in the TEL. 
Coding for er.ch lesson continued for 30 minutes after which the computer 
automatically shut off the coding terminals. Ihe three instructional groups 
were taught simultaneously in three different class^roor-s . > 

A multiple baseline design was selected for the two teaching phases 
of the project. In the baseline condition, trainees taught the lessons 
they had prepared without receiving feedback of any kind. The number of 
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lessons taught without feedback varied from trainee to trainee. The baseline 
teaching always included at least the first three lessons and varied up to 
the first six lessons taught. Assignment of trainees to vaiiable baseline 
teaching trials was random. The second controlled condition was teaching 
after the trainee received computer printout information or feedback on his/ 
her baseline teaching performance and a set of instructions in modular form 
on how to read the computer printout. This feedback was based on both COC- 
STRAT and NIAN-STRAT categories. 

The trainees were also instructed in procedures for using the printout 
information to graph the percent frequency of occurrence of behaviors for 
each of the observation system categories. Ihey were also assigned the task 
of graphing all categories in the baseline period. At this stage the trainees 
weve also required to evaluate their baseline teaching on the basis of their 
graphs. They then selected and listed in order of personal preference three 
of the observation system categories from COG-STRAT and NIAN-STRAT that they 
judged as most important to their teaching and wished to work on for improving 
their teaching performance. Within a week of receiving baseline information, 
the trainees were required to select a single behavioral category for 
subsequent improvement. V.\o final se^lection was arrived at in consultation 
with a staff member. This choice became, in most cases, the category that 
was targeted for improvement in subsequent lessons. The trainee also had to 
indicate whether she/he wanted to accelerate or deccelerate the percent fre- 
quency of occurrence of the target behavioral category in subsequent lessons. 
In most cases the tr- jes first choice was an acceptable behavioral cate- 
gory indicative of a relatively stable baseline performance on that cate- 
gory during baseline. All of these activities were carried out after the 
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baseline period and before the trainees taught with f eedl^ack . 

In addition to these procedures, the trainees were randomly assigned to 
feedback conditions, flalf the trainees were assigned to a printout -only 
feedback group, and half to a scope and printout fcedh.-ick group. Hie scope 
feedback group was presented with a self-instructional module and instruc- 
tions from a staff member on how to interpret CATTS video scope feedback. 
This step was performed prior to the trainees teaching with feedback. During 
the feedback phase, the weekly lessons continued as during baseline .and,in 
addition, all trainees received printouts on their latest lesson within two 
hours after teaching. 'iTie group receiving instantaneous video or scope 
feedback always had a video monitor \x\ their classroom which showed the 
cumulative percent frequency of occurrence of the targeted behavioral cate- 
gory that the trainee had chosen for teaching behavior change. Hence, this 
group obtained immediate feedback on the criterion teaching behavior while 
continuing to teach. The printout-only group was required to graph the 
percent frequency for the targeted category after the printout was received. 
This data was recorded on a cumulative individual graph for the category 
and included the trainees baseline percentages, 'in addition, all trainees 
kept a running record of the percent frequency in category for each successive 
lesson by entering the information for the previous lesson on a graph in the 
labor?.tor>'. 

The instantaneous video feedback group was required to fill out a post- 
teaching questionnaire after each lesson they taught. It was expected that 
the questionnaire would serve to reinforce the trainees attention to the 
scope during teaching. Among the items in the questionnaire was one in which 
the trainee was required to estimate the number of times he looked at the 
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video monitor. Another question required the trainee to reconstruct the 
video feedback display for the entire 30 » minute lesson. Tlie number of 
lessons taught with feedback varied from trainee to trainee. The feedback 
teaching included at least four lessons and varied up to seven lessons 
taught, depending on the number of lessons taught during the baseline phase. 
The teaching phases of tlie project covered a total period of ten weeks. 

Tlie results of the study revealed that all trainees in both treatment 
conditions significantly increased their baseline ^ criterion .rate of per : /;r- 
mance by a ratio of 2.8:1 as a function of CATTS feedback. There were no 
differences between the two CATTS feedback groups .during bpseline on ' eir 
individually chosen criterion measure. However, during treatment, trainees 
in the CATTS scope and printout condition increased their criterion rate of 
'responding to a significantly greater degree (2:1 ratio) than did trainees 
in the delayed printout-only condition. 

Conclusi ons. The results of the present CATTS project have demonstrated. 

i 

the effectiveness of CATTS immediate visual and delayed (printout) feedback 
on increasing critical cognitive and management behaviors of special educa- 
tion teacher-trainees in a controlled laboratory setting. In this project, 
all trainees in both CATTS treatment conditions were able tc r-ignir-icantly 
increase their baseline rate of teaching performance as a function of CATTS 
feedback.. Moreover, trainees who received the CATTS instantaneous visual 
feedback were able to increase their criterion rate of responding to a rela- 
tively greater degree than did trainees who received CATTS delayed or printout 
feedback following their teaching. Hence, these findings once again verify 
the importance of feedback and, in particular, immediate feedback or knowledge 
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of results in human learning and skill acquisitic . Feedback which is trans- 
mitted within seconds of the occurrence of the behavior to be modified appears 
to facilitate greater teacher behavior change than feedback which is delivered 
after the lesson. In our view, successful acquisition of teaching skills in 
any competency or competency-based tru.ining program in special education is 
critically dependent upon [a) specification of "appropriate*' teaching be- 
haviors, patterns, and envi ronments , (b) reliable and valid feedback of 
teaching performance during practice or acquisition trials, (c) immediate 
availability of feedback information to the trainee during the course of an 
'ongoing lesson, and (d) access to in.'^'?rmation on previous performance for 
ipost- teaching analysis and review. Hence, the CATTS system can serve as a 

i 

Iprime vehicle for the discriminati ^ veneration, and evaluation of specific 

I \ 

;teaching\ competencies in exi st i ng to;icher education programs in special and/or 
regular education. In essence, CATTS is a versatile and comprehensive system 
.which can be applied in numerous ways wi.thin the teacher-training field. 
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TABLE 1-.2 ■ . ' 

CATTS OBSERVATION SYSTEMS CODING SCHEDULE 
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TABLE 1.3 TENTATIVE CATTS TRAINING SQIEDULE 

Week #1 



Session ^1 
ilh hours) 

Monday 



Session *2 
(1 hour) 

Tuesday- 



Session -^3 
(1 hour) . 

Wednesday 



Session 
(1 hour) 

Thursday 



Session ^^5 
{IH hours) 



Monday 



1.1 Go over any questions on written script (10 min.) 

1.2 Intro, to DITPsM^: (30 min.) 

1.2.1 Call out pupils 5 categories for 
button box and mental warm-up drill 

1.2.2 Code handed-in examples 

1.2.3 Break (5 min.) 

1.3 Code COG-STRAT Training Tape Seg's 1-40 (40 min.) 

1.4 Assignment ^2: Prepare a 5-min. Lesson relevant 
and appropriate to Steakley's class. Due on 
Session ^2 {5 min.). (Be sure to include all 
categories.) 



2.1 Wanr-up: Code seg^s 1-40 again (30 min.) 

2.2 Role-playing (30 min.) 

2.2.1 1 teacher 

2.2.2 2 pupils 

2.2.3^ 1 or 2 coders 5 trainer/coder 



3.1 Code continuous segments on COG-STRAT 
Training Tape (1 hour) 

3.1.1 Start by coding 5, 6, 7 only, using 

pupil code 1, 2, 9, 0 
3; 1.2 Replay and code pupils only» using 

above codes 

3.1.3 Replay and code "everything 

3.1.4 Continue with seg. #2 (Ibid) 



4.1 Continue coding continuous segments on training 
tape (30 min.) 

4.2 Role-playing (30 min.) 



5.1 Criterion Tape (Segments 1-54) (45 min.) 

5.2 Read printout and determine who passes and 
who does not (answer questions) 

5.3 Replay scj;'3 1-54 for people ivho do not pass 

5.4 Read printout, etc. 



TABLE 1.3 TENTATIVE CATTS TRAINING SGIEDULE 
(Continued) 
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Session #6 
(1 hour) 

Tuesday 



Session #7 
(1 hour) 

Wednesday 



Session ^^8 
CI hour) 



6.1 Criterion Tape (Zuck's lesson), 
twice (45 min.) 

6.2 Read printout 



7.1 Make-up sessions 

or 7.2 Retraining 

or 7.3 Retake criterion 

8.1 Make-up or retrain or retake criteri 



to be coded 



on 



Thursday 
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TABLE 1.4: OBSERVER AGREEMENT GIECKS 



System 


Number of 
Observers 
Involved 


Type of Check 


Agreement 
Coefficient(s) 
Used 


COG-STRAT 


27 


Initial Criterion- 
Related (#1) 


P ; P ' 

0 0 


COG-STRAT 


27 


Initial Criterion- 
Related (#2) 


P ; P ' 

0 0 


MAN-STRAT 


9 


Initial Criterion- 
Related (#1) 


P ; P ' 
o' 0 


MAN-STRAT 


9 


Initial Criterion- 
Related (2) 


P ; P ' 

0 0 


COG-STRAT 


27 


Initial Intra- 
Ob server 




MAN-STRAT 


9 


Initial Intra- 
Observer 




MAN-STRAT 


9 


Maintenance 
Criterion-Related 


P ' 

0 


MAN-STRAT 


9 


Maintenance 
Intraobserver 


P • TT 

°f ^ 


COG-STRAT 


26 


r!aintenance 
Criterion-Related 


p ' 

0 


MAN-STRAT 
COG-STRAT 


i 

1 


Live Maintenance 
Live Maintenance 





1 TABLE i.5 INITIAL CRITERION- RELATED OBSERVER AGREE>ENT: 

\ 

COG-STRAT 







PUPIL IDENTIFICATION 






PUPIL #0 


PUPIL 


#1 


PUPIL 


#2 


OBSERVER 






■ 

p 

o 


o 


P„ 
o 


o 


Jo Ann 


A, 


.93 


.94 


1 .00 


l.GO 


1.00 


.98 




c 


1.00 


1.00 


.95 


.96 


.87 


.91 


f Marv Kay 


f. 


1.00 


1.00 


.95 


.99 


1 .uu 


1 nn 


( Kate 


Z. 


.97 


.98 


on 




1 nn 


1 .nn 


> Joyce 


B. 


.92 


.95 


.95 


.95 


.97 


1-.00 


t Janette 


S. 


1.00 


1.00 


1 , 00 \ 


1 nn 
1 . 00 




QQ 

^y 


7 DeLynn 


T, 


1.00 


1.00 


n c 

.9b 


Q T 

.y / 


,UU 


1 nn 


B Teri 


H. 


1.00 


1.00 


.95 


.97 


1 nn 
l.OU 


1 nn 

1 ^uu 


) Janet 


R. 


1.00 


1.00 


1 ^00 


1 nn 
1 . uu 




97 


D Paula 


S. 


1.00 


1.00 


1,00 


1.00 


.90 


n c 


L Bev 


R. 


1.00 


1.00 


.86 


.88 


.87 


DO 

.89 


I Terri 


P. 


1.00 


1.00 


. 95 


Q 7 


.y^ 


97 


S Barb 


W. 


.92 


.95 


.95 




1 nn 

1 . uu 1 




• Chris 


T. 


.92 


.91 


.90 








S Shelly 


L. 


.89 


.92 


.90 


OA 

.90 


on 
.yu 




^ Susie 


A. 


.92 


.96 


' .90 


.88 


.93 


1 .00 


7 Rueben 


F. 


.95 


1.00 


1.00 


.96 


. 87 


. -Jul 


J Marcy 


S. 


1.00 


1.00 


1 .00 


i *00 


Q 1 

.y^ 




? di'iris 


N. 


.95 


.96 


.90 


.90 


.93 


QQ 

.y y 


p Marilyn 


0. 


1.00 


1.00 


.86 


.88 




Q7 


I CaroJ 


M. 


1.00 


1.00 


.95 


— ■ 


.y .i 


9R 


I Jin 


F. 


1.00 


.99 


.95 




1.00 


1. 00 


S Diane 


V. 


1. 00 


1.00 


1.00 


1.00 


.97 


.98 


i Jo 


S. 


11 .00 


1 . 00 


1.00 


l.GO 


.93 


.99 


» Kathy 


B. 


.95 


.99 


.95 




.87 




i Diana 


M. 


• 86 


. 84 


1.00 


1, -00 


.8.1 


.84 


7 Arlene 


G. 


.97 


1.00 


1.00 


.97, 


. .77 






MEAN 


.969 


.977 


.95;;; 


.956 


.9.^3 


.966 


'*P^* « Percent ;*gree 

P' « Percent agree 
o 


ment ag, 
Enent ag 


ainst criter: 
ainst criter: 


Lon based on raw frequencies 
Lon based on percent frequencies 
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TABU: I.b IMllAL CRITERION- RELATED OSSERVtR AGREEMENT: 



COG-STRAT 

77 





PUPIL IDENTIFICATION 


• 


#3 . 


#4 


OBSERVER 


o 




n 
r 


^0 


1 


.95 


.96 


.96 


.95 


2 




1 

Q Q 




1 nn 

1 .UU 


3 


• 95 


• 99 


.96 


.98 


4 


.90 


.97 


1 .00 


1 .00 


S 


1.00 


.98 


.96 


1.00 


6 


,95 


.99 


1 .C 0 


1 .00 


7 




. 9d 


. 92 


OA 

. 96 


•8 


cc 


1 .uU 


1 nn 
1 .UU 


1 nn 
1 .UU 


9 


1 An 

1 .UU 




. yo 




10 




.98 


A/i 

.9o 


OA 

. 96 


11 


.90 


.96 


.93 


.94 


12 


i . UU 


i .UU 




1 nn 

1 . UU 


13 


. CO 


.94 


. 9o 


. yo 


. M 


1 .00 


1 .00 ' 


n A 

.90 


o-^ 

.92 


15 


-95 


.93 


1 .00 


.99 


16 


. 9U 


1 .UU 


. yo 


1 nn 

l.UU ii 


17 


.95 


A O 

.98 


A*? 

.92 


1 nn 
i .UU 


18 


*91 


.93 


.92 


.95 


19 


1.00 


.93 


.96 


1.00 


_ _ao 




1 .UU 


r yo 


1 nn 

1 .UU 


21 


.76 


.82 


.93 


.93 


22 


1.00 


1 .00 


1.00 


1.00 


23 


-95 


1.00 


.92 


.96 


24 


.95 


1.00 . 




1.00 


' 25 


.,.95 


1.00 


1.00 


-97 


26 


.95 


.99 


.77 


. 1' 


27 


.90 


1.00 


.88 1 


1.00 ■ 


MEAM 


■ -940 

i 


.974 


.949 


.970 



*Pq » Percent agreerent against criterion based or, raw frequencies 
Pq ■ Percent .-jcrecrent arainst criterion ' -'sed on ptfrcent freq''cncies 



ERIC 



TABLE INITIAL CRITERIGS- RELATED OBSERVER AGREEMENT: 

COG- ST RAT 





Ca 


ttegory Identification 


Categi 




Category #2 


Category *3 


OBSERVER 


i— 




P 


Pi 




P' 


1 

2 
•S 

4 

5 

6 

7. 

8 

9 
10 
11 

12 . 

13 
14 
15 
16 
27 
18 

_^ 19 

20 
21 


1 . 00 


1 on 


1 .00 


1.00 1.00 


1.00 


— I. Of) 


.99 




.00 


on 




.90 


.93 


30 


.93 


77 




_ .90 




.91 


.91 


.CI 




1.00 


.98 


.90 


.97 


.83 


81 


.83 


.85 


.83 


.85 


1.00 


1 nn 

1 . Ull 


__ . 83 


.84 


.71 


.72 


.70 


74 


. 90 


.95 


1.00 


..00 


.91 


.92 


On 


.96 


.91 


.91 


.90 


.96 


.91 


.90 


.91 


.90 


.80 


.86 


.91 


.92 


.83 


.84 


.90 


.95 


.83 


.83 


.91 


.91 


.80 


.85 


.90 


.97 


.83 


.81 


.80 


.88 




.92 


1.00 


1.00 


i:oo 


1 .00 




• oy 


.91 


.89 


.50 


.54 


.91 


.83 


1.00 


.97 


.90 


.99 


1.00 


.94 


.90 


1.00 


1.00 


.94 


..91 ^ 


.92 


1.00 


1 .00 


.90 


.95 


.83 


.82 


1.00 


.98 


1.00 


.98 


.91 


.91 


1.00 


1.00 


.83 


.83 


1.00 


.99 




.90 


.90 


.96 ' 


22 


.91 ' .92 


.91 


.92 


1.00 


1 .00 


23 


.90 


.95 


.70 


.74 


.77, 


.78 - 


24 


1.00 


.98 


1.00 


.98 


1.00 


.98 


25 


1.00 


.97 


1.00 


.97 


1.00 


.97 


26 


.S3 


. R7 




.62 


.77 1 .80 


27 


.83 


.76 


.90 


1.00 


.70 j .82 


MEAN 


.913 


.919 


.9C0 


.915 


.870 1 .895 



•Pq » Percent agreer.cnt against criterion based on raw frequencies 

» Percent a;:rccr^r.t afiir-^t criterion based cn or cent frec;ucr.cies 
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TABLE 1.5 INITIAL CRITERIC3N-RELATED OBSER\'ER AGREEMLST: 

COG- ST RAT 



79 





CATEGORY IDENTIFICATION* 




Categorv* 






Category #6 


OBSERVER 


p. 

0 


P' 
o 


p 


Pi 


p 

0 


P' 

e 
















1 


.90 


.90 


1.00 


1.00 


1.00 


.94 


2 


1.00 


.99 


1.00 


l.f-' 


1 .00 


1 .00 




.90 


.93 


1.00 


1.00 


.88 


.86 


A 


1 00 

X • w 


1 no 


.85 


.90 


.92 


.71 




1 .00 


.98 


.85 


.92 


1.00 


,90 


V 


1.00 


1 .00 


1.00 


1.00 


1.00 


1.00 


7 


1.00 


1.00 


1 .00 


.95 


1 .00 


1.00 


8 


1.00 


1.00 


1.00 


1.00 


1.00 


1 .00 


9 


1.00 


1.00 


1.00 


1.00 


.83 


.79 


10 


1.00 


.98 


^ 1.00 


l.OC 


1.00 


.98 


X X 


.80 


.85 


1.00 


1.00 


1 00 1 1 .00 


12 

X c» 


1.00 


1.00 


1.00 


.94 


.91 ' 


.86 


13 


1.00 


.98 


1.00 ■ 


1.00 


.95 


.88 


14 


1.00 


1 .00 


1.00 


.96 


1.00 


.96 


1^ 

x«^ 


.90 


.97 


1.00 


.98 


1.00 


1.00 


16 


i.OO 


.97 


.92 


1.00 


1.00 


1.00 


1 / 


1.00 


.94 


1.00 


.94 


1.00 


1.00 


18 

X V 






1.00 


1.00 


.87 




1Q 


1 00 

X • w 


• :»o 


1.00 


.99 


1 nn 






1.00 


1.00 


.92 


.98 


1.00 


.99 


z 1 


1.00 


.99 


1 .00 


.94 


.95 




22 


.90 


.95 


i.OO 


1.00 


.88 


.87 


23 


1.00 


1.00 


.85 


.89 


.94 


.93 


24 


1.00 


.98 


. 1.00 


.99 


.ns 


.89 


25 


1.00 


.97 


.62 


.67 


1.00 


1.00 


26 


1.00 


1 .00 


.76 


.75 


l.M 


I ' .9^ , 


27 


.80 


.93 


-85 


.98 


1.00 


.95 


MEAN 


.967 


} .972 


.949 


.955 ■ j 


.963 


.927 



*Pq « Percent agreement against criterion based on raw frequencies 

" Percent agreement against criterion br^red on percent frequenciei> 



TABLE 1.5 INITIAL CRITERION-RELATED OBSERVER AGREEMENT: 

COG-STR.^T 



30 





Category Identification 




Catej^ory »7 


Category #8 


Category *9 


OBSERVER 


P* 




P 


P' 


Po 


P' 

— 0^ 


1 


1.00 


1 .00 


.91 


i .91 


.91 


.97 


2 


1.00 


1.00 


1.00 


.99 


.91 


.90 


S 


.85 


.82 


1.00 


1.00 


.91 


.93 


4 


.65 


.61 


1.00 


1 . 00 


.91 


.91 


5 


.73 


.68 


.92 


.89 


.91 


.89 


6 


1.00 


1.00 


1.00 


1.00 


.80 


.83 


7 


1.00 


1.00 


.92 


.92 


1.00 


1.00 


8 


1.00 


1.00 


1.00 


1.00 


1.00 


1.00 


9 


1.00 


1.00 


1.00 


1.00 


1.00 


1.00 


10 


1.00 


1.00 


1.00 


.99 


1.00 


.98 


11 


1.00 


1.00 


.92 


.92 


.90 


.95 


12 


.75 


.75 


1.00 


1.00 


1.00 


1.00 


13 


1.00 


1 .00 


1.00 


.98 


.91 


.89 


14 


1.00 


.98 


1.00 * 


1.00 


i.no 


1.00 


IS 


1.00 


1.00 


.92 


.89 


.83 


.81 


16 


1.00 


.91 


1.00 


.96 


.91 


.88 


17 


1.00 


.93 


.91 


1.00 


1.00 


.94 


18 


.63 


.62 


1.00 


1.00 


.91 


.92 


19 


.88 


.89 


.91 


.99 


1.00 


. .98 


20 


1 on 


'i 00 

± ■1/1/ 


1 . 00 


1.00 


1.00 


1.00 


21 


.88 


.89 


1.00 


.99 


1.00 


.99 


22 


.85 


.81 


1.00 


1.00 


.91 




TS 


.73 


.69 


1.00 


1.00 


.91 


.92 


24 


.85 


.78 


1.00 


.99 


.91 


.90 


25 




.7] 


.01 


1 . '^0 


l.OO 


.97 


26 


i:co 


7. 00 


.92 


.95 


.90 


.92 


27 


.88 


.97 


.82 


.96 


.90 




i.no 


MEAK 


.906 


1 .892 


i , 965 


.975 


.939 


.941 



♦p a Percent arre^^rcnt arninsr criterion br»sed on raw frcnueiicies 
o ^. • . 

Pi Percent a^rocr.ent a*n.i;r critoricn based or. percent r:"equcncie5 
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OBSERVER 


Identification ' 


Catc^c 


»ry *10 


Category *11 


Category ^12 ) 


p. 

o 




P 

n 






2 J 


1 
2 
S 
4 
5 
6 
7 
8 
9 
10 
11 


. 92 




.93 


1.00 j 1.00 


1.00 


1.00 


.64 


.68 


1.00 


1.00 


1 00 


Q T 


1.00 


1.00 


.88 


.85 




. OS 


.91 


.96 


.88 


. 87 


1 . (lO 


.94 


.55 


.59 


.38 


.90 


1 nn 


1 . 00 


.91 


.95 


1.00 


.98 


• oLr 




.64 


.67 


.88 


.87 


on 


o c 

. y5 


1.00 


1.00 


1.00 


i.OO 


1 1 on 


1 nn 


.55 


. .58 


.88 


.87 


92 




.73 


.78 


.88 


.89 




. y 3 


.82 


.36 


1.00 


1 .00 


1 .00 


1, 00 


12 
IS 
14 
IS 
16 
17 
18 

- - 19 
20 

22 
23 

• 

24 

25 


.64 


.67 


1.00 


1.00 


.90 




.55 


.59 


1.00 


1.00 


.90 


.97 


.91 


.95 


. 88 


. 87 


1 . 00 


1 .00 


.82 


.88 


.63 


.64 


1 no 




.73 


.80 


. 75 


7Q 


1 . on 


1 .00 


.46 


.52 


.75 


.80 


1 nn 
1 . Ill; 


on 


.82 


.86 


1 . 00 


1 . 00 


1 on 


1 nn 


.82 


.88 • 


1.00 


1.00 


80 




.82 


.87 


1.00 


1.00 


.90 


.96 


.73 


.78 


1.00 


1.00 


.90 


.96 


.91 


.95 


1.00 


.98 


.90 


.95 


.91 


.96 


1.00 


1.00 


1.00 


1.00 


.46 


.49 


1.00 


1.00 


1.00 


1.00 


.91 


1.00 


1.00 1 


i.nn 


.QO 


oq 


26 


.18 


.19 




. 90 


1.00 i 




27 


.82 


.96 


1.00 


.85 


.80 




MEAN 


.746 


-793 


1 

.934 t 


.928 i 


.943 


.93 j 
.914 1 



•Pq " Percent at;reerent ajraSnst criterion based on raw frcquen ies 
P* * Percent w_:T-f -cnt a^^riin-i: cntcricp. ':rai-Hi on norcent fr?::i:oncies 
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TABLE 1.5 INITIAL CRITEPIO.S'-RELATcD OBSER^-ER AGR£E.yESr: 



ODG-STIl'" S2 









PUPIL IDE.STIFICATIOS' 








PUPIL to 


PUPIL 




PUPIL 


#2 


OBSERVER 








P 

O 




1 




llo Ann 


A. 




.93 


.94 


1 .UO 




^ fin 






S. 




1.00 


1.00 


QC 




.0/ 


Q 1 


Vkry Kay 


Y. 




1. 00 


1.00 




.99 


1.00 


1.00 


rite Z. 


.97 




>gQ 


.92 


i.on 


1 nn 


Joyce 


B, 




.92 


.C5 ! 


.95 


.95 


.97 


1.00 


Jfl^ette 


S. 




1.00 


1.00 ^ 


1.00 


1.00 


.97 


.99 




T. 




1.00 


1.00 


.95 


.97 


1.00 


1.00 


Terl 


H. 




1.00 


1.00 


.95 


.97 


1.00 


li.OO 


Jsnet 


R. 




1.00 


1.00 


1 .00 


1. 00 


.93 


.97 


Paula, 


S. 




1.00 


1.00 


1.00 


2.00 


.90 


.95 


Bev 


R. 




1.00 


1.00 


.86 


-88 


.97 


.89 


Terri 


P. 




1.00 


1 .00 


.95 


.97 


.93 


.97 


Baib 


W. 




.92 


.95 


.95 


.95 • 


1.00 


Q8 


Chris 


T. 




.92 


.91 


.90 


.93 


.97 


QQ 


Shelly 


L. 




.89 


.92 


.90 


.90 


.90 


.9fi 


Susie 


\a. 




.92 


.96 


.90 


.8& 


.93 


1 .no 


itueben 


F. 




.95 


1.00 


1.00 


.96 


.87 


06 


Marcy 






1. 00 


1.00 


1 .00 


: .00 


.93 


.96 


Chris 






.95 


.96 


.90 


.90 


.93 


.99 


Marilyn 


o.\ 




1.00 


1.00 


.86 


.88 


.93 


07 


Carol 


M. 




1.00 


1 .00 


.95 


.96 


.93 


08 


Jia 


F. \ 




1.00 


.99 


.95 


.98 


1.00 


1.00 


Ciane 


V. \ 




1. 00 


1.00 


1.00 


1.00 


,97 


.98 


Jo 


s. \ 




"t 

1.00 


1.00 


i 1.00 


1.00 


.93 


.99 


Kathv 


B. \ 




.95 


.99 


1 .95 


.93 


.87 


QA 


Diana 






.86 


.84 


i 

1 1.00 


1 no 






Arlene 


G. 




.97 


1.00 


1.00 


03 


.77 


RR 


MEAN 


.969 


.977 


.953 

^ - .,— — 


.956 


.933 


.966 



■ Pcrccbt agreement against criterion based on raw frequencies 

\ \ * 

?• ■ Percent agreement against criterion based on percent frequencies 
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TABLE 1.6 INITIAL CRITP.RICN-P^L^riD G3SER\tR AGREE.'-ENT: 

wsj-s'-STPAT 





P'JPIL IDF.vTIFICATION 


DBSERVFR 


PuTtil -'1 'i r-UT/il =1 1 i 


PuTsil ^2 


Fo* 1 ?A j To 




Po 


P ' 


■ i 




.16 I'i .SB 




.92 


.98 


2 


1.00 


1.00 


l.on 


1.00 i 


1.00 


1.00 


5 


.99 


7 ' 

1.00 ' 


.97 


1.00 


i.OO 


.97 


4 


.98 


.99 


1.00 


.99 


1.00 


l,OQ 


5 


. 92 


:: 1" 

.98 ■ 


.97 


.97 


1.00 


.92 


6 


.89 


.96 


,34 


.95 


1 .00 




7 


.89 


1.00 


.81 


.96 


.34 


.95 


8 


.95 


1.00 


• .88 


.99 


1.00 


.91 


9 


.80 


.92 


.91 


.96 


1.00 


.87 
















Mean 


.936 


.979 i 


.918 


.979 


.973 ( .948 










OR<>FRVFR 


i Pu-.il **3 


Pupil -4 


1 




j Po* 




Po 


Po 




















I 


It ■ ■ 
.87 


.93 


.88 


.99 






2 


1.00 


.98 


.94 


.97 






3 


.95 


. ""S 


.88 


.95 






4 


1.00 


.97 


1.00 


.99 






5 


1.00 


.92 


.77 


.86 ^ 






6 


.93 


.82 


.79 


.90 






7 


i.no 


.89 


.88 


1.00 






8 


1.00 


.93 


.68 


.77 






9 


.92 


.94 




1.00 _ 




















Mean 


.963 ! .329 


! ,849 


.937 


1 





♦Po^Percent asreenent against criterion based on raw frecpiencies 
P^=Percent aprcenent at^ainst criterion based on percent frequencies 



04 
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TABLE 1.6 INITIAL CRITERICN-P^LATED OSSEPATR ACREENIEM : 

MAN-STRAT 



CATEGORY IDENTIPKATIOM 



Category t 



Category o 



C2tegor>' 3 





I O 1 




r \j 


PI 






1 




?— 

.89 , 


1 . 1; J 


r- 

. 94 • : 


i 

.90 i 




2 


1.00 ! 


.99 i 


1 .00 i 


1 .00 1 


.96 i 




T 


1.00 i 


.94 j 


1.00 i 


.97 


.97 


.99 


4 


.99 1 


■ i.nn ' 


1.00 i 


1.00 


QQ 


.9C 


5 


.93- 1 


1.00 

t 


1.00 j 


.94 


1 

.98 j 




6 


.81 1 


T- 

.88 i 


r 

1.00 ' 


.94 


1.00 ! 


.94 


7 


.76 i 


.87 ! 


1.00 1 


.88 


.93 


.97 


. 8 


.92 


.99 


^ i 

1.00 ; 


.94 


.99 


.88 


9 


.74 


.86 


1.00 - 1 


.88 


.93 


.95 




1 


i 


1 


.Mean 


-.904 ■ 


.936 


1.00 


.943 


.961 


.956 






Category 4 j 


Category 5 


Category ^ 


OBSERVER 


Po 


Pi I 


Po • 


P' 


Po 














'^1 


.89 


.99 


1 .00 i 


.90 


1.00 


.93 


2 


.89 


.91 


i.no 


.96 


1 .00 


1.00 


3 


.78 


.80 


.88 


.90 


.75 


.73 


4 


\^.00 


1.00 


.90 


.86 


, .8'6 


.86 


5 


.89 


1 

.99 


.88 


.93 


.86 


.SO 


6 




.93 


1.00 


1.00 


.75 


.71 


7 


.90 


.80 


.89' 


.83 


^1 .00 


.89 


8 


.56 


.61 


.90 


.80 


' r 

.50 


.57 


V 


. .44 


.53 


1 i.on . 


. "3 


.86 


.75 




1 ! 






! 


Mean 


! .817 


' . 829 


i .93? 


' ,890 


.842 


.804 



*Po = Percent a^rceinent against criterion based on raw frequencies 

= Percent a^^reement against criterion based on percent frequencies 
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TABL£ i.o INITIAL CRITrRION-RrLATHD Q''<-^:^r.P. AnFrE*>C:M: 

KAN-STRAT 



OBSERVERS 



Cate^orv / 



-ate^orv s 



CateeoT 



PI 



1 


I.nn ■ .93 i l.'^O .94 ! .71 ^ 


2 


.36 ' 1.0^ : 1.^0 .77 • .73 j 


3 


1.00 i .97 1 I. 00 : .97 


.71 


.72 


4 


1.00 i 1.00 i 


1.00 1 1.0'J 

! i. 


1. 00 ' 


1.00 


5 , 


.86 I .80 


.86 j .98 


.77 


.80 


6 


.67 [ .62 i 1.00 j .94 


.77 .82 


7 


.67 .59 , .86 i I.CO 


.94- . 


1.00 


8 


.75 


.71 


1 

.86 i .98 


.59 


.63 


9 


.75 


.65 


.71 1 .37 


.94 


1.00 








) , 

t 

1 






Mean 


.840 1 


.796 


! 

.921 : .964 


.800 


.830 






Categorv in 


Category 11 Cate;;ory 12- 




Po. i Pi 


?o ; ■ 1 Po i 


P ' 








1 


1 


i.no 


.94 


. .77 i .82 1 .33 


.33 


2 


1.00 


I.no 


.85 


.82 


1.00 


1.00 


3 


i.no 


.97 


.77 1 .78 


.57 ^ 


.67 ' 


4 


1.00 1.00 


.77 1 


.76 . 


I. 1.00 


1.0b 


5 


1.00 


.94 


.77 


.82 


.67 


.75 




• i.on 


.94 


.69 


.74 


1.00 


1.00 


7 


.45 


.53 


.92 


1.00 


.67 


.75 


8 ■ 


1.00 


.94 


.62 


.65 ■ 


.67 


.75 


9 


.86 


1.00 


.92 


1.00 


1.00 


.1 .00 




( 




! 
1 






Mean .921 


i .918 ; .7S7 


.821 


i 7fiq ■ -RnA_ __ 



•Po « Percent a^reerenc a:ainsr criterion based on raw freaucnciet 
pf ^ Percent actrecnent: a^iinsc criterion based on percent frecuencies 



TABLE 1.7 COG-STRAT INITIAL INTRA-OBSERVER AGREENENT 



PUPIL IDENTIFICATION CATEGORIES 



OBSERVER 


Pe^ 


D 


t 




Of 




1 Jo Mn A. 


• 26 


.51 


. So 


. 14 


.CO 


. OH 


2 Laura S. 


.24 


. 88 


. 84 


. lo 


.0/ 


fiC 
. OO 


T Karv X. Y. 


.22 


.95 


.94 


.15 


.85 


• 82 


4 Kate Z. 


.21 


.96 


.95 


.16 


.86 


.83 


5 Joyce B. 


.23 


.88 


.84 


.20 


.90 


.88 


6 Jannette S. 


.24 


.84 


.80 ' 


.16 


.87 


^85 


7 DeLynn T. 


.24 


.88 


.84 


.16 


.87 


.85 


8 Teri H, 


.25 


.92 


.89 


.18 


.90 


.88 


9 Janet R. 


.23 


.88 


.84 


.17 


.88 


.86 


10 Paula S. 


.22 


.94 


.92 


.17 


.94 


.93 


11 Bev R. 


.28 


.87 


.82 


.17 


.93 


.92 


12 Terri P. 


.24 


.94 


.92 


.19 


.88 


.85 


IS Barb W. 


.21 


.90 


.87 


.16 


.86 


.83 


14 Chris T. 


.21 


.96 


.95 


.22 


.92 


.90 


15 Shelly L. 


.22 


.90 


.87 


.17 


.85 


.82 


16 Susie A. 


.25 


.94 


.92 


.16 


.85 


.82 


17 Rueben F. 


.22 


.92 


.90 


.16 


.90 


.88- 


18 Marcy S. 


.25 


.95 


.93 


. 18 


.90 


.88 


19 Chris N. 


.22 


.94 


.92 


.17 


.86 


.83 


20 Marilyn 0. ' 


.25 


.88 


.84 


.17 


.90 


.88 


21 Carol M. 


.23 


.87 


.83 


.15 


.87 


.85 


22 Jim F. 


.22 


.96 


.95 


.19 


.80 


.75 


23 Diane V. 


.24 


.86 


.82 


. 19 


.89 


. 00 


24 Jo S. 


• 






19 


.93 


.91 


25 Kathy B. 


.26 


.95 


.93 


.15 


.88 


.86 


26 Diana M. 


i .23 


.86 


.82 


.14 


.77 


.73 


27 Arlene G. 


.23 


.95 


.94 


.17 


! 

1 .88 


.86 



o 

ERIC 



TABLE 1.8 INITIAL INTRA-OBSERVER /'AGREEMEOT : 
JIAN-STRAT 



87 





Pupil I.D. 


j Categories 


OBSERVER 


p 








P -rt . 
Of 


1 


.20 


.83 


.79 


.19 


.82 


.78 


2 


.21 


.79 


.73 


.16 

-i 1 


.84 


.81 


3 


.20 


.84 


.80 


i 17- 


.83 


.80 


4 


.21 


.87 


.84 


.21 


.73 


.66 


c 

D 


.22 


.79 


.73 


.18 


.91 


.89 


6 


.21 


.78 


r 

.72 1 


.14 


.79 


.76 


7 


.21 


.85 


.81 i 


.15 


.81 


.78 


8 


.21 


.89 


.86 i 


i .16 


.77 


.73 


9 


.20 


.92 


.90 I 


.17 


.80 


.76 



I 



93 



TABLE 1.9 MA.INTENANCE INTRA-OBSER\'ER AGREE^^F.NT; 

MAN-STRAT 





Pupil I.D. 


1 Categories 






Pof 


IT 


! 




ir 


1 


.37 . 


-78 


.65 


.21 


.82 


■ .77 


Z 


.35 


.92 ' 


.SS 


1 .15 


.8? 


.87 


3 


.30 


.79 i 


.70 


.15 


.80 


.77 


4 


.41 


.65 


.41 


.13 


.71 


.66 


5 


.36 


.79 


.6/ 


.18 


.63 


.55 


6 


.38 


.87 


.79 


.15 


.82 


.79 


7 


.37 


.65 


.44 


.17 


.79 


.75 


8 


.39 


.79 


.66 


.17 


.82 


.78 


9 


.49 


.90 


.80 


.20 


.89 


.86 



D'3 



89 

TABLE 1.10 MAINm'.ANCE CRITERION -RELATED OBSERVER AGREEMENT:- ' 

MAN-STRAT 





PUJ^IL-IDEXTIFICATION 




0 


2 


3 


OBSERVER 


Po* 




Po 


o 


Po 


P' 
*^o 












1 

















2 


1.00 


.99 


1.00 


1.00 


l.CO 


.99 


3 


.64 


.71 


.91 


1 Art 

1 .00 


.99 


on 
.89 


4 














5 


.88 


.90 


1.00 


.97 


1.00 


.98 


6 


.57 


.69 


.82 


.98 


.97 


.85 


7 


.93 


.98 


.82 


.87 


.95 


1.00 


8 














9 


.45 


.54 


.82 


.98 


.98 


.82 
















Mean 


.745 


.802 


.895 


.967 


.982 


.922 


OBSERVER 




4 




Po* 


PA 




1 








2 


1.00 


.98 




3 


.75 


.60 




4 










.73 


.71 




6 


.63 


.76 




7 


..80 


.76 




8 








9 


.80 


.67. 












Mean 


.785 


.747 





*Po = Percent agreement against criterion based on raw frequencies 

* Percent agreement against criterion based on percent frequencies 
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TABLE 1.10 VAlWrniPHCB CRITERION- RELATED OBSER\'ER AGREE^ENT: 

' MAN-STRAT 





CATEGORY IDENTIFICATION' 


OBSBWET 


! 


2 


3 


Po* i ?^ i! Po f P^ II 


Po I PA 




-1 




' 2 


I. GO 


.."59 i 




1.00 


.99 


3 


.64 


-.72 






.98 


.88 


4 














5 


.88 


.90 






.98 


.96 


6 


.56 


.67 




.93 


.89 


7 


.91 


.96 






.91 


.97 


8 














9 


.44 


■ .S3 






.99 


.84 








j 1 










.738 


.795 


i 


\ 


.965 


^•922 










OBS£R¥EF 


4 


5 


6 


Po* 


pi 


Po 




Po 


P* 


1 




l! 


\ 




■ ! 




i.no 


1.00 i 


! i.no 


1.00 


' 1.00 


.99 


■ 3 


l.€0 


1.00 


.88 


.77 


.70 


.79 


4 














. S 


1.00 


1.00 


' .86 


; .88 






«& 


1.00 


1,00 


.88 


.73 


1.00 


,82 


7 


1.00 


1.00 


.86 


.92 


.90 


.96- 


8 








1 






9 


1.00 


l.'.^O .36 


1 . .96 


.71 


.59 ' 












rtean 


-1.00 


1.00 11- .890 ! .877 


1 .830 


. 800 



*Po « Percent atrreenent against criterion based oit ra^; freouencies 

=» Percent agreerncnt against criterioii based on percent frequencies 

/ 



/ 

/ 

- ; • ■ - - / 
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TABLE 1.10 MMNTENiWCE CRITERICN-RELATED OBSERv'ER AGREEl-lENT: 

HAN'STRAT 





CATEGORY IDENTIFICATION 




i 


8 l! 9 


OBSERVER 


Po* 1 P' i 


Po ! PA fi Po 


PA 










1 




y L 








2 1.00 


1.00 }1 1.00 ! 


.99 ! 


1.00 


.99 


3 


1.00 


.89 


1.00 


! 

.88 


.89 


.81 


4 














S 


.95 


.92 


1.00 


.98 


.73 


.70 


6 


.95 


.87 


! .85 1 .74 


.88 


.95 


7 


.91 


.86 


r ■ ; 

l.OG ■ .95 


.89 


.9b 


8 














9 


.84 


.99 


1.00 


' .83 


.80 


.67 
















MEAN 


.942 


.922 1 


.'982 

• 


I .895 


,865 


.845 




* 






OBSERVER 


10 


f 

11 


12 


Po* 


P' 

1 


; Po 


P* 


Po 




1 




1 










2 


1.00 


1.00 i 


1 1.00 


1.00 


1.00 


1 .,'99 


3 


1.00 


.89 


! .92 


.98 


' .75 


.60 


4 












i 


5 


1,00 


.96 


.75 


.78 


1.00 ii -98 


6 


1.00 


.83 


.83 


.99 


.89 


1 ■ .74 


7 


1.00 


.96 


.89 


.84 


.50 




8 














9 


1.00 


' ' .83- 


..83 


.69 


.50 


!l .(62 






' ! 1 


Ii \ ■ 




I 1 nn ! -912 h .870 1 .880 


1 .773 f. .7m^ 



*Po - I?^'(5rrceair aTrreement against criterion based on rmw |%emi±iijfl:cies 

w Perceint rsj^sement against criterion based on pe-^ccrgm:. rreiquencies 



tmE 1.11 MAINTENANCE -CRITCRIOX-RELATED OBSERVER AGREBIENT: 

COG-STRAT 92 1 







PUPIL IDENTIFICATiaN 


j 






0 


1 


2 


3 


s' 




r 

O 





PA 


n f 


P5 


1 


1.00 


.97 


.89 


.98 


.99 


2 


.89 


.89 


1 .95 


.84 


.92 


- 2 


1.00 


.97 


.99 


.93 


1.00 


. 4 


.88 


.96 


1.00 


1.00 


.99 


5 


.83 


.93 


.89 


1.00 


.96 


6 


.80 


.93 


.85 


.90 


.•92 


7 


.92 


.94 


.98 


.92 . 


.97 


8 


.87 


.97 


.97 


1.00 


.99 


9 


1.00 


- 1.00 


.96 


1.00 


.99 


10 


.93 


.90 


1.00 


.94 





11 


.89 


.95 


.89 


.97 1 


.98 


-12 


1.00 


l.GO 


.88 


.95 


.95 ' ... \ 


IS 


.98 


1.00 


.93 








M 


.70 


.87 


.87 


"37 


IS 


— — 






£i) 


.81 


.93 


.95 \ 


I.J0O 1 .?i8 


-177 


.92 


.8* 


.92 ' 




1.100 






.99 


.90 


.96 1 




.^97 






.94 


.98 


.96 [ 


.91 i 


.96 






.90 


.93 


.92 ' 


\.m \ 


.,99 






.88 


.90 1 


.89 1 


.sre i 


1 :oo 




22 


.74 


.92 1 


.94 ! 


.m ' 


.95 




r 


.94 


1.00 


.86 1 


l.OiO 1 


.98 




24 


.90 


.87 ! 


.81 1 


.74 1 


L.iOO 




25 




i.nn ' 


.97 1 




07 


-1 


:26 


.79 


.92 


.84 1 


1.0Q> ! 


:99 




27 


.80 ! 


.86 i 


.95 i 


l.OQ) 


1- 00 




MEAN 


.894 


.936 


.924 


.943 


-974 





« Percent agreement against criterion based on percent frequencii^s 
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TABLE 1.11 MAINTESANCE CRITERICV- RELATED OBSERVER ACREETIENT: 

COG-STRAT 





CATEGORY IDENTIFICATION 




1 


2 


3 


4 


5 


- 

. fi 


OBSERVER 






Po 








\ 

1 


-90 ! .65 1 .5 i _0Q 


.87 




2 




.73 i .93 


.99 


.77 




- 3 . 


• 95 


.94 


.92 


1.00 




OR 


4 

5 
6 
7 
S 

9 
10 

11 

is 


X • \J\J 


.90 


.92 


, .95 


.«;? 






. 91 i . 74 




7n 




1 1 00 


.90 i .87 


.24 


I .79 


..8.T: 


\ 99 


.84 \ .T\ 


.67 '! j' 35rt- 


1.00 


.84 


.84. 




1*00 




..■71 ' I .81 ij 


•93 




a.0C- ...95 


.8? 11 










t OO 


,?0 


7c i T? _ 


•urn 


•95 


.76 


.47 


-72 




.94 


.76 


. 84 II . 83 


. 82 









!|l 








QQ 


.72 !■ .90 ! .58 


• 74 




1 on 


.92 ! . .93 


.'64 


.99 




1.00 


i .94 i .88 


.95 


.94 


i.cm 


19 
20 

-21 
22 

- 23 
24 

25 

26 
27 

MEAN 


.94 


i .84 i .67 


.61 


' 1.00 


^.^ 


1-00 


.90 ' .97 


.97 


i.no 




1. 00 


.94 


.85 1 


.60 


.81 




1.00 


.87 


.75 


.60 


_ ^ 




1.00 


.79 


.65 ll .85 


.70 j 




1.00 


.72 


' 111 


.95 


l.rnn. 


.69 


.98 ; l.CiO fl -97 1-00 




.96 


.84 1 .SI .97 .93 


» ©8 


1-00 


-85 .87 " .92 ' .59 




.952 


.844 


j 

.8ie 1 .81G 


.832 


1 

1 

1 



m Percent agreement against criterion based cn percent frequencies 



TABLE 1.11 MAINTENANCE dRITERION-RELATED OBSERVER AGREEMENT: 

COG-STRAT 



94 







CATEGORY 


IDENTIFICATION 






7 


8 


9 


10 


OBSERVER 


pf * 

o 


PA 
o 


P' 




1 


-74 


.98 


l.Otfli 


.77 


2 


.90 


.35 




.79 


3 


.87 


.117 


.:94!. 


.54 


4 


1 ..78 


.-78 


.977 


.99 


5 


UjCO 


.£3.5 


l.OB 


.91 


6 


..T9 


.S>3 


.944 


.90 


7 


.71 


.m 


1 ,00) 


-89 


8 


^98 


..?32 


..96S 


.74 


9 


^90 


.236 




.40 


10 


.89 


.m 


-981 


.83 


11 


.94 




..9S 


.98 


12 


.95 


.m 


..91 


,55 


13 


.90 




..817 


.54 


U 


.91 


,98 


l.OO 


.99 


15 




— — 








16 


.76 


l-ClO 


1.00 


.88 


17 


i 1.00 


-89 


1.00 


.70 


18 


1.00 


.92 


1.00 


.85 


19 


.70 


.79 


.99 


.71 


20 


.92 


.94 


1.00 


.70 


21 


.90 


.99 


.99 


.78 


22 


.97 


.97 


.95 


.35 


23 


.86 


.90 


1.00 


.97 


24 


1.00 


.93 


.96 


.72 


25 


1.00 


.99 


.95 


.77 


26 


5.80 


.93 


1.00 


l.CO 


27 


.95 


.88 


.97 


1.00 


MEAN 


.889 


.921 


.970 


.796 



ERIC 



■ Percent agreement against criterion based on percent frequencies 
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PILOT STUDIES 
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PILOT STUDY I 

The Effect of CATTS Feedback on the Question Behavior 
of Teacher-Trainees in a Real Class Instructional Setting 



October 26, 1973 
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PILOT STUDY I 

The Effect of C^TTS Feedback on the Questioning Behavior of Teacher- 
Trainees in a Real Class Instructional Setting. 
Introduction 

This pilot study of the effect of CATTS feedback was carried out in 
conjunction with the practicum phas^e of a course in methods of teaching 
educable mentally 'retarded children. The students enrolled in the course 
were mainly I» U. juniors, majoring in the education of the mentally 
retarded. 

The aim of the study was to assess the baseline frequency of use of 
specific teaching categories of behavior by trainees in a real classroom 
situation, and then measure change'5 in xhe use of these same categories of 
behavior under various feedback conditions. 

Utilizing trained obsein^ers to record the cognitive demands (levels 
of questions) and behavioral control strategies of the teacher- trainees, a 
baseline of performance was obtained for each trainee. During the feedback • 
phase, the observers siiailarly recorded the teaching behaviors of the trainees 
engaged in classroom teaching. The purpose of the study was to demonstrate 
and evaluate the effect of CATTS feedback (instant feedback) on .increasing 
the number of high-level questions asked by the teacher during a given lesson. 

In order to sensitize teacher-trainees to an awarenesis and understanding 
of different levels of questioning that can be generated in an instructional 
interchange with pupils, the trainees underwent a ten-hour training program 
in the Individual Cognitive Demand Schedule (ICDS) observational system. 
The system was developed by Lynch and Ames (1971) , and is based in part on 
the theory of instruction proposed by Gagne^ (1967) . It is a hierarchical 
system that categorizes questions asked by teachers into various high and 

Er|c 103 • ' 



xviff x^wxd* iiic;. xun — v^awc^j^v/x xc;d xji^xuuc; axm^xc; vcxuax uc^uicuiud uiAau ' 

require, for example, habitual responding, observing, recitation of previously 
learned materials and remembering. Higher level categories include questions 
that require explaining, defining, classifying, applying and comparing, and 
inferential and problem solving responses. *Also included in the higher 
level classifications are questions requiring value judgment and "make-believe'* 
responses. 

In all, the study entailed four major phases: First, participation by 
trainees in observation system training; second, measurement o£ trainees' 
cognitive demands in the classroom; third, measurement of trainees* 
cognitive demands in the classroom under CATTS instant, delayed, or no- 
feedback conditions;, fourth, measurement of trainees' cognitive demands with 
no feedback, to determine the maintenance of behaviors obtained under feedback- 



1. Observation bystem Training . j * 

The teaching behaviors seen as critical to the establishment of a 
successful teaching environment by the teacher fall into two broad 
categories: behavioral management strategies and pedagogical strategies 

> /■'..■■ 

manifested by cognitive demands initiated by the teacher. Observation- 
coding systems are probably the most reliable technique for objectiye 
measurement of those classroom behaviors deemed important in any given 
approach to instructional method. Rosenshine S Furst {197 Z) , and Flanders 
(1970) have extensively discussed the advantages of observation systems as 
a reliable, low-inference measure of classroom behaviors. 

Two observation instruments were used to record the classroom behaviors 
of trainees and pupils: The Indiana Behavior Management Scale (IBMS) was 
used to record a wide range of off-task behaviors manifested by pupils, and 
, the teachers' response to these behaviors. The IBMS, developed by Fink and 
Semmel (1971), categorizes pupils' behaviors into classifications such as 
types of verbal or physical manifestations of off-task behavior, self-involve 
ment, .'ind whether the behaviors are aggressive or interactive in nature. 
Teacher-control responses are categorized into eleven classifications of 
types of control statements. 

The data derived from the IBMS was expected to result in a profile of 
individual and group patterns of pupil off-task behavior, including percent- 
'aae of off-task behavior in the total instructional period, frequency of 
off -task behaviors that occur during instruction^ and frequency of types of 
control measures invoked by teacher- trainees in response to pupil off-task 
behaviors. ^ ■ 

The pedagogical strategies displayed by teacher-trainees were observed 
and coded in terms of the types of questions asked of pupils. The ICDS 
(Lynch and Ames, 1971) was used to record and describe classroom behavior in 



the cognitive domain. Tlie teacher-pupil categories that are subsumed in the 
system were described earlier in the report (page 97). Tliere are four low 
and seven high-lovel coj;nitive demands, as well 'i\r^ two miscellaneous and 
four feedback categories. The data derived from this system should yield 
a profile of frequency and percentage of occurrence of any category over the 
entire instructional period. 

C oder training . Training procedures for each of the observation systems 
were developed independently and involved different personnel. A pool of 
approximately ten trained IBMS observers were available at CITH, and it con- 
sisted of center personnel who had previously mastered the system. Each 
coder had undergone intensive training sessions, and most of the IBMS coders ' 
were also experienced coder-trainers. All had criterion test scores that 
ranged from .79 up to .93. A criterion test was administered during the 
fourth week of the project as a check of maintenance of coder reliability. 
During the la^t two weeks of the project, three students from the K580 
methods course who had received IBMS training in ^connection with other 
methods course activity, and who had criterion test scores of .80 and above, 
were selected as coders for the project. 

There were fewer trained ICDS coders, available at CITH, so that three 
. CITH staff members were specially trained as .ICDS coders during the week 
prior to initiaj data collection. Tlirough consensus-training workshop- 
sessions, all coder trainees were able to achieve criterion test scores of 
.80 and above. There were six different ICDS coders' used throughout the 
data' collection period. A reliability maintenance test was administered 
twice during data collection and once immediately following the completion \ 

of the study, / 

In addition to mastering the system to a reliability score of above 
.80, each ICDS coder underwent training to criterion in ''button „box" coding. 



Both systems used in the present study are normally arranged so that 
coders pencil-check letter symbols representing each category on data 
sheets. For automation of the ICDS coding, an on-line button box (4x4 
with ten^numbered buttons, resembling a TOUCH -TONE telephone face) was used. 
Through ^transfer of training from letter symbols to numbers, each ICDS 
coder was able to observe and button-code each observation. The coded 
data generated was transmitted by telephone cable to the TEL computer and 
processed. 

' All 33 practicum students participating in the classroom phase of 
the CATTS feedback study were arbitrarily divided into three groups for^ 
observation system training. There were four two-and-one half to three-hour 
training sessions over a two-week period. 

. ^ "The instruction was centered around a pre-instructionaj. package 
developed at the Center for Innovation in Teaching the Handi^capped • (CITH) . 
The materials for observation training included a programmed training manual , 
a workbook and several audio and video tapes for instruction and practice. 
Each session was conducted by an instructor who had previously been through 
full ICDS training and who had also participated in two retraining workshop* 
sessions, during which an abridged version was used for instruction of 
teacher-trainees in the ICDS system. 

Training in ICDS included introductory remarks by course instructors 
on the nature of observation systems and their relevance to teaching skills, 
a review of all ICDS teacher categories, role-playing, and practice coding 

from audio- and video-taped materials. Criterion tests were administered 

' • • ' ^, 

*The workshop sessions were organized and conducted by Karen Greenough, 
whose efforts are much appreciated. 



during the last hour of the third session and the beginning of the fourth 
session. Although coder training vJas not the purpose of these observation 
system training sessions, the criterion tests were used to assess the degree 
of accuracy with which the student trainees. were able to discriminate the 
various categories of classroom questions/ ' 



1 1 . ^ e^suring baseline teaching l ^J^}^ j ■ " 

Subjects and class organization . The 33 methods class students who 
participated in the ICDS training were scheduled into a classroom teaching 
session to conduct a fifteen-minute lesson. The classes utilized for 
teaching were those sponsored by the University Affiliated Facility (UAF) , 
a residential/educational unit for EMR children, ages 7-12. There are 
three UAF clrp^ses located in Building 8 of the f .U. 'Jnivcrsity School complex. 
Tlxe CATTS L -..-ic/rl lation of the CITH Teacher Hdu ja.tL^on Laboratory (TliL) i s 
located in BuiJi ,'ing 7 of the same University Schv oi facility. 

For the duration of the project, all lessons were taught between 10:30 
and 11:30'^A-M. Each day, pupils were randomly assigned to one of the three 
classrooms. , • • . ' 

Three consecutive fifteen-minute lessons were taught each day,- with a. / ■ 
five-minute break between lessons. Each class contained a maximum of seven 
pupils. I 

Lesson assignmonts' ^ Each teacher-trainee received a lesson topic out- 
line twenty-four-hours before the scheduled teaching period. The lesson 
topic assignments were developed in cooperation with the head teacher 'at* the 
UAF, and were chosen to augment the ongoing language arts program. An effort 
was made to establish topical continuity in the lessons taught consecutively. 

The lesson topic outline included a listing of primary objectives, the 
purpcfite of the lesson, suggested approaches and terminal goals (desired 
outcome of lesson). Teacher-trainees were encouraged to be creative in 

lesson planning. It was noted that the approaches suggested in the outline ' ■ 

/ ' -. . . ■ 

-Were for . assistance in planning, but were not to be interpreted too rigidly. . 
Among topics included were, such items' as vocabulary building,, and size and 
spatial relationships associated with three concepts. Also*^ included were 
compound words', pi^fi-'^es , classification and logical grouping and vowel sounds 



III. Experimental phase , 

Qyerview of study > llie first two weeks of the project were used to 
cycle all thirty-three student trainees through ten hours of instruction in 
discriminating the categories of the Individual Cognitive Demand Schedule 
(ICDS) . During the third week these same situdent traiaees were scheduled 
to teach one fifteen-minute .lesson. The data gathered from this initial 
teaching contact was used to describe the baseline of teaching performance 
obtained prior to any experimental intervention. Following completion of 
the baseline data phase, each trainee taught three more lessons, under one 
of three feedback conditions. Twenty-seven of the thirty-three trainees were' 
randomly assigned to one of three groups. Each gr6up received a different 
feedback condition: instant f,b, (CATTS monitor),' delayed f.b,- (computer - 
printout after the lesson), and no feedback. The remaining students partici- 
pated in the baseline and maintenance teaching session, but did not teach 
the three lessons on which feedback was controlled. 

The major objective of the study was to demonstrate the effect of feed- 
back oh the trainees* ability to increase, specific: types of questioning be- 
havior. All of the trainees had as their goal the increase of the n ibex* of 
high-level questions asked during the lessons they taught. The cate^{ories 
of questions which they were. asked to increase were in ferring , problem - 
solving , defining- classifying a nd applying-comparing . The instant feedback 
displayed on the video monitor, therefore^ showed the ^current cummulative 
frequeticy of questions in each of the four cognitive, categories. 

There was no feedback given during the fifth lesson taught by trainees. 
The "data gathered during this last instructional session was used to determine 
the maintenance of the teaching behaviors in each of thie feedback groups. 



.. . / 

Behavior management workshop . There were se\^QT^l faetdhl; affecting the/ 
deAci^(On. isiscrtjK^teinpose a workshop in classroom beha^^tor man{^ -j^vHt Ifechniques 
betW:tc;«V. ths uYtiitial teaching week (baseline) and trhe first^ e3S]?erimentEl week 
First the edue^s^tJonal program at the UAF relies heavily on a program of 
behav 'tt modi :tl cad: ion . The teacher-trainees in the proj^s^cr -:;md no previous 
ekpc ;ie2nc'<o wifr;. m e technique and many had little knowledge} f i\t , DurinK 
the Cxjrsc wee?: j: teaching, the trainees were briefed o:i the. Aor 
Modification ririchnique prior to their entering the clasnr^^ssgn- f -oject 
assistants deisuxibed only the rudiments of the prograTii, e.g^, ^ t.y^f-.^ mechanics 
of appropri.ately marking "in control'' ox "out of contrOV'"' mhec*);^- on ''control*' 
sheets taped t& the children's desk, and the role of poi/ative n negative : 
•reinforcement in management of behavior. ; Secondly^ the ^.meheT'^rainees 
in the present pTOgram were mostly inexperienced"*^ in the cll^ssroc ni or in group/ 
instruction. . For many, this was their first experience t^ic^-^m a classroom 
situation, and some encountered' behavior man agerneni^ prolbiesM im their first 
attempt at teaching. In addition, the problc ri'^uiiv o... *?r>^ lack of 
management experience or skills ^were exacerbated ^y the exrtremely heterogeneous 
nature of the learning and behavioral characteris^tics of the UAF children. ^ 
According to the head, teacher at the UAF, several of the pupils were, persistently 
hyperactive apd volatile, a few had impaired hearing or vision , and a fev; 
were markedly withdrawn. In all, about two-thirds of the ipupils had serious 
behavioral problems. / = 

Data-gathering procedures and organization of the teaching practicum 
under controlled feedback conditions . A substantial effort was required to 
coordinate the coupe rat ion of many separate systems that were necessary for the 
realization of the study. For example, each lesson taught by a trainee was 
prescheduled, ^ouid lesson topics were assigned and ready, twenty -fo.ur hours 



p^iOT to the lesson. Lists of the daily randomization of pupils were made 
available to the head teacher of the UAF, so that random assignment of pupils 
to classrooms was completed before the first teachi^/r-trainee: arrived. Six 
trained observation system coders- were required for . each session.. The coders 
were briefed daily on names of teachers, pupils, nature of lesspm, etc., 
prior to their assignment to coding stations. Tlic teacher- traiae^es were 
briefed prior to their entering the classroom to teach. PYi:it)a^r technical 
syscems'^ (e.g., CATTS feedback monitor, transmission of rcfiW.,h-:, to sxmfnater , 
printout for delay f.b. group) described elsewhere ir ■ '^Vi^ ^^^nTtrrt , wef€ i 
ograration during each teaching session. Back-up te:^'±z.czul i,y:-j.ems, laue as 
au;?Jiliary audio and video taping of the sessions, - aJiso iKcorrrvorsit^^^ in 
the total operation of the data-g:athering* proced^ Continc^ency >,iTro: -«ilures 

were also established in anticipation of a possib;c breakdown of aay m the 
above systems. ^ , . 

Coding' procedures . At least fifteen minutes prior to each instructional 
session, six coders were present at the Teacher Education LaboTati)ry (TEL), 
and were briefed on daily procedures-: Each coder was assigned (by prescfeedliled 
randomization) to a classroom. There was one ICDS and one IBMS coder foT 
each classroom. Th*ey were given sufficient data sheets for three consecutive 
fifteen -minute observations, and also received copies of the lesson out line 
and the names of pupils in each room for that day. After synchronizing 
watches, coders entered the observation rooms. 

The IBMS coders had the additional task of signaling th'e waiting, teacher- 
trainee to enter the classroom. A flashing red light, on the "button box*' 
indicated that coSing transmission would begin in two minutes* After two 
minutes, a red light again flashed on the ''coding device" and both; ICDS and 
IBMS coders began recording observations; the ICDS observer on the- button 



box, the IBMS observer usin^ data sheers. UCI): data shcctfj; wqvo also avnii- 
able to >the coder, in case of system failuixe. /Ifter f i f'taen minutes tme 
IBMS coder entered the classroom to ! npri'?il ^vir ' r.nclirr thar the session .: is 
over. A five-minute break fol ; .)weil., J""^ ^hv-; Tin sec ;)nd -'n^'^Sicc^on !^e)»am 

The procedures for the se - :m:^ iv -d ..i^ fe^...f raci ;^ *''"<a] xtv ->[i.or\s \>^:^(^t^ t-uc 
same as for the first, v}'(}' ' ^ j-^'-'Ur'^^^'^d hr : v^e? !: fSion^, 

Coders returned to v : : ;^ :i upun •(y\'^\iplf^^ :.jiOv. oi la^^t ley- '\{:r[.,.^ and ^ll:'y 

icheckcftd and completed data snircts Kb i^li , filed aioK^ig vrj^rm trire ^im^ititer 
tape omtput for each lesson. 

Teacher-trainee role . The studr-nt trr-. -^ee,^ were brier'^ed on the i^^spon- 
sibilities and requirements for conductiiaj; -acit lessons dir-ing a grc ;^ 
meeting and through written comrmunicatioay^., . 

In addition to information conc/rrTiin}.* sche^/ialiivg and i:^)^'.:^^^^^ routines, 
students were apprised of th^ natiure of tl y ^Jnboristtory schoaa (UAF) , the 
behavior modification prograin, and the rr'n^es and characteriiiatics of children 
.with ispecial problems . . 

iBefore their first lesson, trainees in ^he immediate f .b. group were 
told Ithat the. frequency of occurrxence of any of thn four target question 
^categories (Def ining-Classifying., Applying-Comparing Inferring, Problem^- 
Solving) would be flashed on the TV monitor in their classroom. The teacher 
entered the classroom upom signal from the IBMS coder, spent from two to a 
maximuin of five minutes in acclimatization and introductory matters, and 
then commenced with the prepared lesson. 

Feedback procedures . Each t^rainee was a' ' ed to set as a goal the in- 
crease in the number of high-level questions asked during a lesson: Tlie>.* 
were asked to try to generate increases in foui" 5peci fic categories: Dcvfiaing- 
■ Classifying . . Applying-Comparing, Inferring, andi PProblem-Solving. 
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llie iainfej^'y t'.b. group taui^-tit in a cliissroom eiquipped with a TV monitor. 
The freafi^jiic ' t-^i jiT vrcurrence of any one of the four larget categories was 
f laslie^d^ ^to tehe -T^^^^^ The number Slashed repr^esantisd the cummulative fre- 

quency^ .r:9Jfii tji'^^^ the lesson b^ r^aan. The teacher could, therefore, glance 

up at the monit^^T and see how m^r^Y questions each of the four types she 
had this far ixskv :. I-igare 5 r r tlic video display configuration print 
as the classroom "oni tor. 

The students in the delay rjiM).ack group had a computer printout of 
frequencv^.^.?. and percentages of cu.i ICDS categories generated during instruc- 
tion ava^;Tir :'e them within ar, ivQMr after teaching. During a group meet- 
ing of all studeTi^ttj> lin the delay i\b. group, pric^rr to the experimental phase 
of the project, an: :ri.iSistant expl-ained the symbols and information available 
on the printout sheext. In addition, a referencta srheet for guidance in print 
out interpretation *was given to each student iiin tiXxe delay group. The print- " 
out sheet listed aJLl ICDS categories that occurred within a five -minute 
interval. There we^e three such five minute intcirvals . A summary of the 
frequencies and percentages obtained during the full fifteen-minute period 
was also included in the printout. ' . - 

The no-feedback group was e^ncouraged to try to increase the number of 
high-level questions but did not receive any information about the fre- 
quency of occurrence of the four high-level question types they were work-' 
ing toward increasing. The printouts were ma de"" available to this group upon 
completion of the last teaching" session . 

M aintenance of teaching behaviors . The fifth and last teaching session 
in the project was conducted in the same manner as the first session* There 

V ■ ' ■ . 

was no feedback of.any type for any of the trainees* The purpose of this 
session w*iis to see whether any of the increases expected in the groups receiving 
feedback ^s^otild be maintained in the absence of feedback of information. 
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^ PILOT STUDY II 

The Effects. of CATTS- Feedback on the Question 
Behavior, of Teacher-Trainees in a Simulation Classroom Experience 
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In design, format and procedures employed, pilot study II was 
essentially similar to pilot study I, The critical difference however, 
was that in the present study, student trainees role-played EMR children 
as a substitute for real children during the experimental (GAITS Feedback) 
phase. 

Tlie purpose of this study was to determine whether CATTS was an 
effective training method when trainees instruct peers instead of children. 
The creation of a simulated classroom through role-playing was expected to 
provide teaching experiences and an opportunity to develop teaching skills 
when children and real classroom settings are unavailable to the trainee. 

Overview 

All ,of th'3 trainees in this study were juniors majoring in special 
education and members of an EMR methods class. They v:'ere random^.y selected 
from the class to participate in the study. 

The design of the study provided that all trainees would teach five, 
fifteen -minute lejsons. The trainees were divided into-three f=^xperimental 
groups; an immediate feedback group a delayed feedback group (in the 
form of a computer printout), and the no-feedback group. The initial 
teaching session, provided the baseline data, the next three teaching 
sessions were treatment phases, and a final session was held to determine 
the maintenance of behaviors. 

The study utilized a simulated setting in which trainees role-played 
educable mentally retarded children. The trainees assumed the roles 
prescribed for them in a written script. These scripts sttpplied information 
on the emotional, social, physical, and academic characteristics of an 
EMR child. Trainee peers were asked lo portray the roles as they perceived 
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them^from the informatiop on the scripts. (See sample script, Appendix) 
In the baseline phase of the simulation study, the children in the University 
Affiliated Facility (UAF) , a residential educational center for primary EMR 
and intermediate children, were taught by the teacher-trainees. During the 
three treatment lessons, teaching took place in the simulated HMI^ class- 
room setting. In -the maintenance phase. UMK classes at the UAI* were used 
again. 

Trained observers coding the TCDS and IBMS systems recorded the cogni- 
tive demands and behavioral control strategies of all the trainees in each 
stage of the study. - 

The specific purpose of the study was to df^monstrate the effects of 
CATTS immediate visual feedback on increasing the level of the trainee's i."^ 
cognitive demands (questions) in a fifteen-minute teaching session. It 
was hypothesized that those trainees receiving the immediate visual- feedback-' 
(CATTS) would produce more high-level cognitive demands than trainees re- 
ceiving either delayed feedback (printout) , or trainees receiving no feed- 
back. 

A workshop was instituted to better acquaint the trainees with the cort-; 
ceputal as well as the practical aspects of the Individual Cognitive Demand' 
System. During the workshop the theoretical basis of the ICDS was pre- " 
sented and explained in detail. The trainees were given several opportu- 
nities to conceptualize the different levels of demands (high and low) by 
improvising high- or lew-level cognitive demands. Role-playing was utilized 
to give the trainees practice in using the ICDS. Each trainee was asked to 
assume the role of teacher, coder, and student- 

In sum. there were four phases of the CATTS simulation study; first, 
the training of teacher-trainees on the ICDS observation system; second ' 

' 122 . 
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a baseline ICDS measurement of the trainees questioning behavior with the 

EMR children, thirdly, the experimental phase in which CATTS immediate 

visual feedback, delayed feedback, and: no feedback were provided in a 

simulated class setting, and fourthly,; the maintenance phase (with EMR 

children), in which no feedback was provided in order to determine whether 

the levels of the questioning behavior obtained under the various feedback 

conditix^ns were maintained. • 

Observation System Training - 

All the teacher-trainees participated in a one-day observation system 

training workshop. The workshop was an intensive session designed to 

acquaint the practicum trainees with the Individual Cognitive Demand System. 

The workshop began with all participants reading an abridged version of 

the ICDS training manual, containing only the definitions and a brief 

explanation of each category. The trainees were instructed to read the 

abridged training manual until they felt they had an. understanding of the 

ICDS categories. They were then required to write definitions for each 

ICDS category and their abbreviations. Each trainee was subsequently 

given a slip of paper with an ICDS category on it and instructed to generate 

the -questions or demands indicated. This activity was continued until 

the. trainees could identify the question types by category. It was expected 

> that constant repetition of ICDS categories in simulated situations 

(role-playing) would be the most effective means of category "retention. Thus, 

the teacher-trainees role-played for several hours. Each trainee assumed 

the role of teacher; student, and observer (coder) . After each role-playing 

session a discussion was held. Role-playing continued until trainees ,felt 

that they could generate examples of specific cognitive demands given by 

either the trainer or their peers. Trainees were then^'asked to simulate^ 

a specific classroom within a given setting and with a given lesson topic. 
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Included in the lesson were specific cognitive demands to be made in the 
five-minute lesson simulation. Every trainee participated in' the simulated 
class experience. Discussions were held after these sessions and -definitional 
problems discussed. At the end of the training sessions oral quizzes were 
administered in order to measure the degree of accuracy of discrimination 
of the various categories of the ICDS. The trainers of the^teacher-trainees- 
were skilled observers who had participated in several weeks of observer 
training on the ICDS. All'had obtained above .80 reliability as ICDS coders. 

B aseline Measurement of Teachin g Behavior: 
S imulated Subjects and Classroom Organizatio n 

The teacher-trainees, in the baseline phase of the study taught in the 
University Affiliated Facility primary and EMR intermediate classrooms. 
There were three classrooms, with a total of twenty-one children, located 
in Building 8 of the University Elementary School complex. In the three 
treatment phases the classrooms used for simulations were located in the 
Teacher Education Laboratory. Fourteen different EMR roles were assigned 
to the role-playing trainees each day. The roles were in script form and 
gave detailed descriptions . of the physical, social and emotional, and 
academic characteristics of a ficticious EMR child. The trainees were 
asked to portray the roles as they perceived them. Simulators role-played 
for an hour each day for four weeks enabling the' teacher-trainees to com- 
plete the three treatment phases. ■ 

Behavior Managem ent Workshop 

The teacher- trainees participated in a behavior management workshop 
prior to the baseline teaching session. Since most of the trainees did not 
have any previous experience with educable mentally retarded, children, , it 

-^ , ' '124 ' ' .. ■ ■■ • 
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was aiiticLpatcd that the trainee would need some r.u'ulel ines in classruom 
management. 

The purpose of the workshop was to ^ive the trainees specific tech- 
niques on behavior management. Problem-solving techni(|ues for specific be- 
havior problems were presented by a task analysis method. The trainees 
were urged to carefully consider any behavioral problem situation and to 
try to give more positive feedback. Allowing for individual differences 
in children was strongly emphasized. 

In order to give trainees an overview of the types of behavioral con- 
trols used by the UAF teachers, the two classroom teachers spoke at the 
workshop. They explained the behavior modification system used and Provided 
examples of when to use positive or negative controls. Video tapes of the 
EMR children in their usual classroom environment were shown. The teachers 
described the »'in controI*7"out of control" sheets and procedures to be 
used by the teacher- trainees in the classroom. In addition to the informa- 
tion on behavior controls, the UAF teachers gave a brief description of the 
childrens* home background. and academic problems. 

The last phase of the workshop concentrated on specific behavioral 
controls that could be used with the children. One of the trainers wtio had 
five years experience. with EMR children raided this discussion. It was 
pointed out that voice changes as a controlling measure are effective in 
instituting "on-task" bffehaviors in the deviant children. Emphasis was 
placed on the positive aspects of controlling of f-task behaviors . Trainees- 
were urged to be more probing and emphatic-sympathetic in dealing with off- 
task behaviors. . , ^ 

L esson Plan Assi ^ gnments 

The curricvlum topics that were used in this study were arrived at by 
discussions held with the head teacher' of UAF. . The lesson plans were a 
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continuation of the curriculum areas that were stressed in UAF classrooms. 
Lesson plans were given to the teacher-trainee 24 hours before the assigned 
teaching period. The lesson plans format was behaviorally based. The 
format began with behavioral objectiveis to be stressed in the lesson. The 
purpose of the lesson, various possible approaches, and terminal behavioral 
goals were included in each lesson. Topics that were covered in the curriculiim 
were science, perceptual skills, oral and written language and listening 
skills. The lessons^^ere constructed so that there was a conceptual frame- 
work from which the trainee could proceed. The lesson ssignments encouraged 
individual creativity in planning by the teacher-trainee. 

Experimental Phase 

The twenty-four trainees participating in the experimental phase were 
randomly placed in three equal groups.. One group received immediate visual 

r 

feedback. The' second group received delayed feedback in the form of a 
computer printout, and the third group received no feedback- of any type. The 
objective of the experimental phase was to demonstrate the effectiveness 
of visual feedback, delayed feedback, and no feedback on the teacher -trainees' 
ability to. increase specific types of cognitive demands. The goal for all 
trainees was to increase the frequency of four ICDS categories: value-judging 
inferring, defining-classifying and app lying-comparing. 

Those trainees in the immediate visual feedback group viewed the TV 
monitor display which showed the frequency tally of the^ four ICDS categories 
that were to be increased by the trainees. The trainees in the delayed 
feedback group received feedback' in the foinn of a computer printout,, within 
2 hours of their.assigned teaching session. The printout showed the 
frequency tally of all thirteen ICDS categories and a summary of the four 
specific categories, to be increased. The third group of trainees received 
no feedback of any kind. 

Inn • - 
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Coding Pr o cedures 

Coders were randomly assi^ed on a daily basis to classroom observation 
rooms. There were three classrooms being obsen^ed and two observers coding 
in each. There was one ICDS and one IBMS coder in each observation area. 
Coder scheduling was randomized so that no two observers would code con- 
sistently together. - 

All /coders met 15 minutes prior to the initial scheduled teaching 
session.'^ They were then given coding packets which consisted of a daily 
class list, lesson plans to be taught by trainee, trainees names, paper 
coding booklets and room number assignment. Stop watches were given to all 

observers as well as the computer operator, and these were synclironized in 

1 - ■ - 

order to 'assure that everyone began coding at the same time. 

Observers then went into observation rooms and started an audio as 

well as a video recorder machine. The computer operator then flashed a 

pne-minute warning signal on a button box. This reminded the IBMS observer 

to go to the classroom and tell the trainee to* begin the teaching session. 

Then the IBMS and ICDS observers began coding. After fifteen minutes of 

continuous coding by the IBMS and ICDS coder, the teaching session terminated 

The iBMS coder then went to the classroom and signaled the teacher-trainee 

to terminate the teaching session. The coders then had five minutes between 

sessions to check all technical hardware f-rudip and video tape recorder, 

button boxes) . The second and third teaching sessions employed the same 

fxoding procedures as the first teaching session. 

Teacher-Trainee Role ^ , 

Trainees arrived at the TKL fifteen minutes before their assigned 
teaching time. They were given a ppxket which contained a class list 
of the children in their classroom, room number, and behavioral control. 
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sheets for the behavior modification system normally used by the classroom 
teachers. Trainees were then briefed on any changes in the classroom, 
i.e., if a child was sick or if there had been any emotional upsetting 
experience earlier that morning for any child. Trainees were also informed 
of options available if children went out of control and needed to be 
removed from the classroom. They were also told that they would have 
behavior control support from a crisis teacher. - " 

Trainees in. the immediate visual feedback group were reminded to 
look at the CRT daring the experimental phase of the study- Trainees 
in the delayed feedback group were reminded that they could pick up their 
delayed feedback forty-eight minutes after they taught their lesson. 

The teacher-trainees went to their assigned classrooms five minutes 
before their teaching time to get acquainted with the children. The->IBMS 
coder then signaled the trainee to start the teaching session. The teaching 
session terminated when the IBMS observer opened the classroom door. 

Feedback Procedures • 

The I CDS observers collected data by entering observations on the. 
button box connected to the computer. / 

Each trainee's goal was to increase the frequency of occurrence of the 
four ICDS categories (VJ, AC, IN, DC) during the fifteen-minute teaching 
.session. Thus, the teacher- trainee concentrated upon generation of high- 
level cognitive demands. 

As the teacher-trainee taught the lesson during the experimental 
phase, a coder entered the observation by entering numbers on the button 
box. The real-time data processing continuously updated the CRT display. 
Thus, each time a trainee generated a demand in either 'of four specified 
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categories, it was displayed on the CRT unit. The trainee then was able 
to see the frequency of high-level demands he nade while teaching. Trainees 
who received delayed feedback also had as their goal the increase of high- 
level demands, as did trainees who received no feedback. Within two hours 
after the termination of th^, teaching session these trainees obtained 
printouts of the codes of their lesson. The delayed feedback, in a form 
of a computer printout, showed frequencies and percentages of ICDS categories 
generated by the trainee during the fifteen-minute teaching period. 
Trainees in this group received a reference sheet to the printout which 
^explained the number tallies and category summaries. The printout ^ad the 
data summarized in three five-piinute segments. All thirteen categories of 
the ICDS used by the trainee were summarized by 'frequency and percentage. 

Those trainees in the no-feedback group were encouraged to generate 
higher level demands during their teaching session, but never received any 
information on the frequency of occurrence of the. higher level demands that 

-V 

they generated during their teaching sessions. However, this group had an 
opportunity to review video tapes of their teaching experiences at. the end 
of the study. . . 

Maintenance of Teaching Behaviors 

The maintenance lesson was the fifth time that the trainees taught 

r 

and was identical to the baseline or first lesson. The same procedure, 
lesson plans and coding techniques were implemented. The trainees did 
not receive any type of feedback but were encouraged to generate higher 
levels demands (DC, AC, VJ, IN). The purpose of the baseline lesson was 
to see if any of the increases expected in the groups receiving feedback 
would be maintained in the absence of feedback. 
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CATTS V Schedule: 



Monday : 



Tuesday: 



Wednesday: 



Thursday: 



Friday:* 



Teaching Tines for Groups X, Y, Z 



Session 



A 9:30-10:00 

B 10:30-11:010 
C i:00-l:30 
U 2:00-2:30 
E 1:00-1:30 
F 2:00-2:30 
G 9:30-10:00 

H 10:30-11:00 

I 12:45-1:15 



PILOT SITJDY RI-FRRn.NChS: 
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